[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:33:09 dist-test-slave-n3tq krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:33:09 dist-test-slave-n3tq krb5kdc[395](info): set up 2 sockets
Jun 19 04:33:09 dist-test-slave-n3tq krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 04:33:09.913113 388 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.011s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:33:10 dist-test-slave-n3tq krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:33:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:33:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:33:10.067719 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33103
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:10.384012 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:10.384298 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:10.384382 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:10.391906 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:33:10.392019 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:10.392068 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:10.392119 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:33:10.392160 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:33:10.401929 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:10.403167 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:10.404418 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:10.415998 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:10.416122 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:10.416030 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:10.416565 408 server_base.cc:1034] running on GCE node
I20260619 04:33:10.417448 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:10.418874 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:10.420068 408 hybrid_clock.cc:648] HybridClock initialized: now 1781843590419985 us; error 73 us; skew 500 ppm
I20260619 04:33:10.420435 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:10 dist-test-slave-n3tq krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:10.426364 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:33:10.427774 408 webserver.cc:460] Webserver started at http://127.0.97.62:32979/ using document root <none> and password file <none>
I20260619 04:33:10.428639 408 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:10.429159 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:10.429443 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:10.432304 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "f4f3a4374ad343749aa5bc968823e935"
format_stamp: "Formatted at 2026-06-19 04:33:10 on dist-test-slave-n3tq"
server_key: "762b64771ffa9c4533f3a34bb0cbe079"
server_key_iv: "274122bd88cc1d53d307fff3f4a71ef2"
server_key_version: "encryptionkey@0"
I20260619 04:33:10.433040 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "f4f3a4374ad343749aa5bc968823e935"
format_stamp: "Formatted at 2026-06-19 04:33:10 on dist-test-slave-n3tq"
server_key: "762b64771ffa9c4533f3a34bb0cbe079"
server_key_iv: "274122bd88cc1d53d307fff3f4a71ef2"
server_key_version: "encryptionkey@0"
I20260619 04:33:10.437989 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 04:33:10.441318 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:10.442641 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:33:10.442790 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "f4f3a4374ad343749aa5bc968823e935"
format_stamp: "Formatted at 2026-06-19 04:33:10 on dist-test-slave-n3tq"
server_key: "762b64771ffa9c4533f3a34bb0cbe079"
server_key_iv: "274122bd88cc1d53d307fff3f4a71ef2"
server_key_version: "encryptionkey@0"
I20260619 04:33:10.442955 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:10.457098 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:10.461169 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:10.461418 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:10.477322 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33103
I20260619 04:33:10.477344 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33103 every 8 connection(s)
I20260619 04:33:10.478608 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:33:10.482631 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:10.483258 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 408
I20260619 04:33:10.483557 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:33:10.485991 388 external_mini_cluster.cc:1428] Setting key 5c014e5d35d0b66f19d989619ae1ca53
I20260619 04:33:10.494545 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935: Bootstrap starting.
I20260619 04:33:10.497572 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:10.498518 475 log.cc:824] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:10.501724 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935: No bootstrap required, opened a new log
I20260619 04:33:10.510303 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3a4374ad343749aa5bc968823e935" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33103 } }
I20260619 04:33:10.510748 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:10.510867 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f3a4374ad343749aa5bc968823e935, State: Initialized, Role: FOLLOWER
I20260619 04:33:10.511502 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3a4374ad343749aa5bc968823e935" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33103 } }
I20260619 04:33:10.511724 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:33:10.511857 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:33:10.512010 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:10.514106 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3a4374ad343749aa5bc968823e935" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33103 } }
I20260619 04:33:10.514580 475 leader_election.cc:302] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4f3a4374ad343749aa5bc968823e935; no voters:
I20260619 04:33:10.515165 475 leader_election.cc:288] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935 [CANDIDATE]: Term 1 election: Requested vote from peers
Jun 19 04:33:10 dist-test-slave-n3tq krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:10.515264 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:10.517956 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:33:10.522002 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935 [term 1 LEADER]: Becoming Leader. State: Replica: f4f3a4374ad343749aa5bc968823e935, State: Running, Role: LEADER
I20260619 04:33:10.522604 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3a4374ad343749aa5bc968823e935" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33103 } }
I20260619 04:33:10.525825 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4f3a4374ad343749aa5bc968823e935" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3a4374ad343749aa5bc968823e935" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33103 } } }
I20260619 04:33:10.527081 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4f3a4374ad343749aa5bc968823e935. Latest consensus state: current_term: 1 leader_uuid: "f4f3a4374ad343749aa5bc968823e935" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3a4374ad343749aa5bc968823e935" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33103 } } }
I20260619 04:33:10.527554 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935 [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:10.527720 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935 [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:10.528096 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:10.490409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43234 (local address 127.0.97.62:33103)
0619 04:33:10.491255 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:10.491270 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:10.492600 (+ 1330us) server_negotiation.cc:407] Connection header received
0619 04:33:10.502068 (+ 9468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:10.502094 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:10.502584 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:10.503004 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:10.506238 (+ 3234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:10.508278 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:10.509905 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:10.510958 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:10.511138 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:10.517079 (+ 5941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:10.517087 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:10.517101 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:10.517138 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:10.520209 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:10.521002 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:10.521008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:10.521014 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:10.521184 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:10.521617 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:10.521621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:10.521623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:10.522139 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:10.522463 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:10.523312 (+ 849us) server_negotiation.cc:299] Negotiation successful
0619 04:33:10.523680 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":376,"threads_started":1}
I20260619 04:33:10.529901 489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:33:10.534835 489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:33:10.543148 489 catalog_manager.cc:1347] Generated new cluster ID: 3276662c1b6149b49041aeb03c13be6e
I20260619 04:33:10.543258 489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:33:10.556273 489 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:33:10.557560 489 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:33:10.566699 489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4f3a4374ad343749aa5bc968823e935: Generated new TSK 0
I20260619 04:33:10.567468 489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 04:33:10.604311 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33103
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:10.925870 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:10.926126 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:10.926246 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:10.933216 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:10.933317 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:10.933462 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:33:10.943222 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33103
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:10.944530 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:10.945910 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:10.958426 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:10.958446 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:10.958429 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:10.959229 501 server_base.cc:1034] running on GCE node
I20260619 04:33:10.959784 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:10.960675 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:10.961864 501 hybrid_clock.cc:648] HybridClock initialized: now 1781843590961821 us; error 33 us; skew 500 ppm
I20260619 04:33:10.962157 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:10 dist-test-slave-n3tq krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:10.965102 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:33:10.966454 501 webserver.cc:460] Webserver started at http://127.0.97.1:44879/ using document root <none> and password file <none>
I20260619 04:33:10.967085 501 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:10.967176 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:10.967417 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:10.970193 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "6eea9cda78164b1fab7a0c788a9680ab"
format_stamp: "Formatted at 2026-06-19 04:33:10 on dist-test-slave-n3tq"
server_key: "982f5ce83dc22fe0cbd00db57621d99b"
server_key_iv: "d8a2e4d236fb951b26eca9e4aa580a5d"
server_key_version: "encryptionkey@0"
I20260619 04:33:10.970901 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6eea9cda78164b1fab7a0c788a9680ab"
format_stamp: "Formatted at 2026-06-19 04:33:10 on dist-test-slave-n3tq"
server_key: "982f5ce83dc22fe0cbd00db57621d99b"
server_key_iv: "d8a2e4d236fb951b26eca9e4aa580a5d"
server_key_version: "encryptionkey@0"
I20260619 04:33:10.975716 501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 04:33:10.979446 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:10.980870 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:33:10.981003 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "6eea9cda78164b1fab7a0c788a9680ab"
format_stamp: "Formatted at 2026-06-19 04:33:10 on dist-test-slave-n3tq"
server_key: "982f5ce83dc22fe0cbd00db57621d99b"
server_key_iv: "d8a2e4d236fb951b26eca9e4aa580a5d"
server_key_version: "encryptionkey@0"
I20260619 04:33:10.981176 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:10.999516 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:11.003633 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:11.003862 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:11.004757 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:11.006428 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:11.006502 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260619 04:33:11.006590 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:11.006629 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:11.038115 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35427
I20260619 04:33:11.038156 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35427 every 8 connection(s)
I20260619 04:33:11.039340 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:33:11.044090 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 501
I20260619 04:33:11.044297 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:33:11.045035 388 external_mini_cluster.cc:1428] Setting key b20576c217e805cae1fa279f5c0bf3b1
Jun 19 04:33:11 dist-test-slave-n3tq krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 04:33:11.060633 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:11.041647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41105 (local address 127.0.97.62:33103)
0619 04:33:11.042024 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:11.042029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:11.043007 (+ 978us) server_negotiation.cc:407] Connection header received
0619 04:33:11.045266 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:11.045270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:11.045340 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:11.045435 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:11.047163 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:11.047844 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:11.050034 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:11.050446 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:11.050547 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:11.054621 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:11.054629 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:11.054634 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:11.054670 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:11.057727 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:11.058386 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:11.058392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:11.058395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:11.058533 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:11.058889 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:11.058894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:11.058898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:11.059238 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:11.059379 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:11.059993 (+ 614us) server_negotiation.cc:299] Negotiation successful
0619 04:33:11.060214 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":190,"threads_started":1}
I20260619 04:33:11.061376 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:11.042205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33103 (local address 127.0.97.1:41105)
0619 04:33:11.042855 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:11.042897 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:11.044943 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:11.045703 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:11.045712 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:11.046316 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:11.046967 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:11.046990 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:11.047997 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:11.048003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:11.049893 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:11.049907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:11.050650 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:11.050656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:11.050909 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:11.051575 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:11.051608 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:11.053905 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:11.057853 (+ 3948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:11.057859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:11.057874 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:11.058270 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:11.058623 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:11.058626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:11.058628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:11.058786 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:11.059344 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:11.059350 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:11.059648 (+ 298us) client_negotiation.cc:769] Sending connection context
0619 04:33:11.059859 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 04:33:11.060230 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":250,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 04:33:11.062857 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33103
I20260619 04:33:11.063174 629 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:11.063843 629 heartbeater.cc:511] Master 127.0.97.62:33103 requested a full tablet report, sending...
I20260619 04:33:11.065943 440 ts_manager.cc:194] Registered new tserver with Master: 6eea9cda78164b1fab7a0c788a9680ab (127.0.97.1:35427)
I20260619 04:33:11.067394 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 04:33:11.081809 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33103
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:11.396859 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:11.397127 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:11.397225 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:11.404053 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:11.404167 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:11.404309 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:33:11.414057 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33103
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:11.415299 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:11.416491 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:11.428712 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:11.428759 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:11.428782 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:11.429555 635 server_base.cc:1034] running on GCE node
I20260619 04:33:11.430150 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:11.431042 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:11.432235 635 hybrid_clock.cc:648] HybridClock initialized: now 1781843591432198 us; error 31 us; skew 500 ppm
I20260619 04:33:11.432566 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:11 dist-test-slave-n3tq krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:11.435359 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:33:11.436645 635 webserver.cc:460] Webserver started at http://127.0.97.2:46589/ using document root <none> and password file <none>
I20260619 04:33:11.437273 635 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:11.437378 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:11.437640 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:11.440284 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "6dfc24f5c5ab4a52a1c09a47714c7d3c"
format_stamp: "Formatted at 2026-06-19 04:33:11 on dist-test-slave-n3tq"
server_key: "37087c4795e6ef8db7be5ae084544b28"
server_key_iv: "7dd58ca358607aab4986ad252047524c"
server_key_version: "encryptionkey@0"
I20260619 04:33:11.441028 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6dfc24f5c5ab4a52a1c09a47714c7d3c"
format_stamp: "Formatted at 2026-06-19 04:33:11 on dist-test-slave-n3tq"
server_key: "37087c4795e6ef8db7be5ae084544b28"
server_key_iv: "7dd58ca358607aab4986ad252047524c"
server_key_version: "encryptionkey@0"
I20260619 04:33:11.445763 635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 04:33:11.449391 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:11.450580 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:33:11.450716 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "6dfc24f5c5ab4a52a1c09a47714c7d3c"
format_stamp: "Formatted at 2026-06-19 04:33:11 on dist-test-slave-n3tq"
server_key: "37087c4795e6ef8db7be5ae084544b28"
server_key_iv: "7dd58ca358607aab4986ad252047524c"
server_key_version: "encryptionkey@0"
I20260619 04:33:11.450886 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:11.467466 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:11.470973 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:11.471220 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:11.472039 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:11.473250 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:11.473323 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:11.473436 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:11.473500 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:11.508380 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43767
I20260619 04:33:11.508419 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43767 every 8 connection(s)
I20260619 04:33:11.509613 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:33:11.517948 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 635
I20260619 04:33:11.518143 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:33:11.518656 388 external_mini_cluster.cc:1428] Setting key 1d22566dbfccc5a79d9470caae7e6102
Jun 19 04:33:11 dist-test-slave-n3tq krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:11.529354 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:11.511990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53691 (local address 127.0.97.62:33103)
0619 04:33:11.512181 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:11.512186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:11.513362 (+ 1176us) server_negotiation.cc:407] Connection header received
0619 04:33:11.515433 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:11.515437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:11.515501 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:11.515588 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:11.517009 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:11.517766 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:11.519688 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:11.520097 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:11.520216 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:11.523550 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:11.523554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:11.523558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:11.523587 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:11.526518 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:11.527162 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:11.527167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:11.527171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:11.527318 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:11.527667 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:11.527672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:11.527674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:11.527972 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:11.528114 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:11.528769 (+ 655us) server_negotiation.cc:299] Negotiation successful
0619 04:33:11.529013 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 04:33:11.530128 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:11.512553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33103 (local address 127.0.97.2:53691)
0619 04:33:11.513209 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:11.513246 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:11.515195 (+ 1949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:11.515780 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:11.515787 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:11.516258 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:11.516846 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:11.516863 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:11.517908 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:11.517913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:11.519571 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:11.519582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:11.520309 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:11.520317 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:11.520586 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:11.521219 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:11.521243 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:11.523364 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:11.526641 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:11.526648 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:11.526660 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:11.527042 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:11.527406 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:11.527409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:11.527412 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:11.527566 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:11.528102 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:11.528107 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:11.528385 (+ 278us) client_negotiation.cc:769] Sending connection context
0619 04:33:11.528660 (+ 275us) client_negotiation.cc:240] Negotiation successful
0619 04:33:11.529033 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":254,"threads_started":1}
I20260619 04:33:11.531442 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33103
I20260619 04:33:11.531703 763 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:11.532353 763 heartbeater.cc:511] Master 127.0.97.62:33103 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 04:33:11.533676 440 ts_manager.cc:194] Registered new tserver with Master: 6dfc24f5c5ab4a52a1c09a47714c7d3c (127.0.97.2:43767)
I20260619 04:33:11.534454 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 04:33:11.553517 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33103
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:11.868141 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:11.868402 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:11.868547 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:11.875815 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:11.875913 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:11.876029 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:33:11.885380 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33103
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:11.886574 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:11.887791 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:11.899899 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:11.899927 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:11.899866 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:11.900688 768 server_base.cc:1034] running on GCE node
I20260619 04:33:11.901227 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:11.902062 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:11.903254 768 hybrid_clock.cc:648] HybridClock initialized: now 1781843591903204 us; error 43 us; skew 500 ppm
I20260619 04:33:11.903525 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:11 dist-test-slave-n3tq krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:11.906279 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:33:11.907934 768 webserver.cc:460] Webserver started at http://127.0.97.3:38859/ using document root <none> and password file <none>
I20260619 04:33:11.908591 768 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:11.908707 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:11.908994 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:11.911664 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "e7bf592da1e34606a008d8dc4c8f9846"
format_stamp: "Formatted at 2026-06-19 04:33:11 on dist-test-slave-n3tq"
server_key: "0218867fafd5bcfa29bf3da024f86c6e"
server_key_iv: "19fad34bc4d69512df439298b62034b5"
server_key_version: "encryptionkey@0"
I20260619 04:33:11.912385 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e7bf592da1e34606a008d8dc4c8f9846"
format_stamp: "Formatted at 2026-06-19 04:33:11 on dist-test-slave-n3tq"
server_key: "0218867fafd5bcfa29bf3da024f86c6e"
server_key_iv: "19fad34bc4d69512df439298b62034b5"
server_key_version: "encryptionkey@0"
I20260619 04:33:11.917098 768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 04:33:11.920274 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:11.921491 768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 04:33:11.921622 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "e7bf592da1e34606a008d8dc4c8f9846"
format_stamp: "Formatted at 2026-06-19 04:33:11 on dist-test-slave-n3tq"
server_key: "0218867fafd5bcfa29bf3da024f86c6e"
server_key_iv: "19fad34bc4d69512df439298b62034b5"
server_key_version: "encryptionkey@0"
I20260619 04:33:11.921805 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:11.942555 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:11.946255 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:11.946489 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:11.947369 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:11.948976 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:11.949051 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:11.949163 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:11.949213 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:11.978983 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39961
I20260619 04:33:11.979025 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39961 every 8 connection(s)
I20260619 04:33:11.980155 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:33:11.987481 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 768
I20260619 04:33:11.987655 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:33:11.988025 388 external_mini_cluster.cc:1428] Setting key 2832ac5585ff96d00395178a0ed24644
Jun 19 04:33:11 dist-test-slave-n3tq krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:11.999011 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:11.982609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44171 (local address 127.0.97.62:33103)
0619 04:33:11.982835 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:11.982842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:11.983962 (+ 1120us) server_negotiation.cc:407] Connection header received
0619 04:33:11.986225 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:11.986229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:11.986296 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:11.986380 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:11.987847 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:11.988680 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:11.990504 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:11.990929 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:11.991033 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:11.994265 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:11.994270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:11.994274 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:11.994305 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:11.996723 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:11.997288 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:11.997291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:11.997294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:11.997402 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:11.997711 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:11.997714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:11.997716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:11.997949 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:11.998051 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:11.998622 (+ 571us) server_negotiation.cc:299] Negotiation successful
0619 04:33:11.998775 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 04:33:11.999960 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:11.983124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33103 (local address 127.0.97.3:44171)
0619 04:33:11.983800 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:11.983842 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:11.985977 (+ 2135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:11.986571 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:11.986580 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:11.987075 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:11.987677 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:11.987702 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:11.988841 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:11.988846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:11.990387 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:11.990400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:11.991141 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:11.991150 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:11.991425 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:11.992050 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:11.992070 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:11.994073 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:11.996818 (+ 2745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:11.996824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:11.996837 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:11.997207 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:11.997473 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:11.997476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:11.997479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:11.997631 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:11.998034 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:11.998040 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:11.998333 (+ 293us) client_negotiation.cc:769] Sending connection context
0619 04:33:11.998544 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 04:33:11.998912 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":250,"threads_started":1}
I20260619 04:33:12.001291 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33103
I20260619 04:33:12.001556 896 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:12.002285 896 heartbeater.cc:511] Master 127.0.97.62:33103 requested a full tablet report, sending...
I20260619 04:33:12.003646 440 ts_manager.cc:194] Registered new tserver with Master: e7bf592da1e34606a008d8dc4c8f9846 (127.0.97.3:39961)
I20260619 04:33:12.004372 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44171
I20260619 04:33:12.015481 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 04:33:12 dist-test-slave-n3tq krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843592, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 04:33:12 dist-test-slave-n3tq krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843592, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 04:33:12.056353 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:12.041772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55748 (local address 127.0.97.1:35427)
0619 04:33:12.042308 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:12.042317 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:12.042678 (+ 361us) server_negotiation.cc:407] Connection header received
0619 04:33:12.042833 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:12.042839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:12.043034 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:12.043167 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:12.044534 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:12.045498 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:12.046740 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:12.047259 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:12.047369 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:12.050973 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:12.050984 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:12.050999 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:12.051030 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:12.053766 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:12.054374 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:12.054381 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:12.054383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:12.054490 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:12.054842 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:12.054848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:12.054851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:12.055169 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:12.055436 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:12.055712 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 04:33:12.055951 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":239,"threads_started":1}
W20260619 04:33:12.057576 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55748
I20260619 04:33:12.070299 629 heartbeater.cc:503] Master 127.0.97.62:33103 was elected leader, sending a full tablet report...
Jun 19 04:33:12 dist-test-slave-n3tq krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 04:33:12 dist-test-slave-n3tq krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 04:33:12.096809 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:12.083823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55754 (local address 127.0.97.1:35427)
0619 04:33:12.084049 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:12.084055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:12.084296 (+ 241us) server_negotiation.cc:407] Connection header received
0619 04:33:12.084520 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:12.084524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:12.084614 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:12.084720 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:12.085960 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:12.086638 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:12.087796 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:12.088188 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:12.088301 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:12.091966 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:12.091972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:12.091976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:12.092009 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:12.094737 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:12.095319 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:12.095322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:12.095325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:12.095436 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:12.095750 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:12.095753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:12.095755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:12.095991 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:12.096154 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:12.096369 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 04:33:12.096590 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 04:33:12.098168 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 501
I20260619 04:33:12.109378 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:12.099086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55768 (local address 127.0.97.1:35427)
0619 04:33:12.099256 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:12.099260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:12.099482 (+ 222us) server_negotiation.cc:407] Connection header received
0619 04:33:12.099600 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:12.099603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:12.099654 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:12.099728 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:12.100781 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:12.101454 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:12.102605 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:12.103002 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:12.103083 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:12.105150 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:12.105155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:12.105158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:12.105192 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:12.107346 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:12.107899 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:12.107903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:12.107905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:12.108014 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:12.108261 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:12.108264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:12.108266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:12.108519 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:12.108679 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:12.108937 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 04:33:12.109075 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 04:33:12.110074 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:12.190721 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 501
I20260619 04:33:12.203701 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 635
Jun 19 04:33:12 dist-test-slave-n3tq krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 04:33:12.218219 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:12.204557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55570 (local address 127.0.97.2:43767)
0619 04:33:12.205013 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:12.205020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:12.205045 (+ 25us) server_negotiation.cc:407] Connection header received
0619 04:33:12.205101 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:12.205108 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:12.205329 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:12.205455 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:12.206520 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:12.207477 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:12.208657 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:12.209186 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:12.209286 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:12.212968 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:12.212980 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:12.212995 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:12.213025 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:12.215722 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:12.216322 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:12.216332 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:12.216335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:12.216538 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:12.216839 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:12.216842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:12.216844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:12.217150 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:12.217379 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:12.217683 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 04:33:12.217864 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":216,"threads_started":1}
I20260619 04:33:12.219581 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:12.299527 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 635
I20260619 04:33:12.317781 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 768
Jun 19 04:33:12 dist-test-slave-n3tq krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 04:33:12.332769 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:12.318603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54650 (local address 127.0.97.3:39961)
0619 04:33:12.319175 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:12.319183 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:12.319213 (+ 30us) server_negotiation.cc:407] Connection header received
0619 04:33:12.319273 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:12.319279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:12.319481 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:12.319611 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:12.320749 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:12.321781 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:12.323035 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:12.323565 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:12.323656 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:12.327548 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:12.327559 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:12.327574 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:12.327609 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:12.330329 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:12.330915 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:12.330921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:12.330924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:12.331041 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:12.331307 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:12.331310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:12.331312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:12.331623 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:12.331841 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:12.332113 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 04:33:12.332301 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":277,"threads_started":1}
I20260619 04:33:12.334036 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:12.414201 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 768
I20260619 04:33:12.427294 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 408
I20260619 04:33:12.428421 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:12.527710 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 408
2026-06-19T04:33:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3605 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:33:14 dist-test-slave-n3tq krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:33:14 dist-test-slave-n3tq krb5kdc[917](info): set up 2 sockets
Jun 19 04:33:14 dist-test-slave-n3tq krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260619 04:33:15.202700 388 mini_kdc.cc:121] Time spent starting KDC: real 2.638s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260619 04:33:15.406306 388 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.110s user 0.003s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260619 04:33:15.564589 388 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.158s user 0.002s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/test-user.keytab.
W20260619 04:33:15.907624 388 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.343s user 0.002s sys 0.001s
Jun 19 04:33:15 dist-test-slave-n3tq krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:33:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:33:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
W20260619 04:33:16.610651 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.62: real 0.665s user 0.000s sys 0.007s
I20260619 04:33:16.611986 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35843
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:16.926429 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:16.926705 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:16.926828 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:16.933898 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:33:16.934000 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:16.934068 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:16.934151 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:33:16.934207 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:33:16.943737 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:16.945053 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:16.946316 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:16.957326 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:16.957355 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:16.957412 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:16.958684 930 server_base.cc:1034] running on GCE node
I20260619 04:33:16.959460 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:16.960837 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:16.962025 930 hybrid_clock.cc:648] HybridClock initialized: now 1781843596961982 us; error 32 us; skew 500 ppm
I20260619 04:33:16.962368 930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:16 dist-test-slave-n3tq krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:16.970000 930 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:33:16.971539 930 webserver.cc:460] Webserver started at http://127.0.97.62:42417/ using document root <none> and password file <none>
I20260619 04:33:16.972221 930 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:16.972349 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:16.972682 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:16.975639 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "517ec619de434b00900db7f21b827af1"
format_stamp: "Formatted at 2026-06-19 04:33:16 on dist-test-slave-n3tq"
server_key: "099d707ce0d1246115b442687020d344"
server_key_iv: "4cd66daed2bbd886fdd584ee163f2363"
server_key_version: "encryptionkey@0"
I20260619 04:33:16.976369 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "517ec619de434b00900db7f21b827af1"
format_stamp: "Formatted at 2026-06-19 04:33:16 on dist-test-slave-n3tq"
server_key: "099d707ce0d1246115b442687020d344"
server_key_iv: "4cd66daed2bbd886fdd584ee163f2363"
server_key_version: "encryptionkey@0"
I20260619 04:33:16.981491 930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 04:33:16.984859 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:16.986195 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:33:16.986363 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "517ec619de434b00900db7f21b827af1"
format_stamp: "Formatted at 2026-06-19 04:33:16 on dist-test-slave-n3tq"
server_key: "099d707ce0d1246115b442687020d344"
server_key_iv: "4cd66daed2bbd886fdd584ee163f2363"
server_key_version: "encryptionkey@0"
I20260619 04:33:16.986531 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:17.000870 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:17.005108 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:17.005369 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:17.022210 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35843
I20260619 04:33:17.022231 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35843 every 8 connection(s)
I20260619 04:33:17.023579 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:33:17.026626 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 930
I20260619 04:33:17.026844 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:33:17.027606 997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:17.027658 388 external_mini_cluster.cc:1428] Setting key 23b75a56cafb0e4b3f9e68425a0af96e
I20260619 04:33:17.041291 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1: Bootstrap starting.
Jun 19 04:33:17 dist-test-slave-n3tq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:17.044545 997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:17.045610 997 log.cc:824] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:17.048177 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1: No bootstrap required, opened a new log
I20260619 04:33:17.049695 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:17.029619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41224 (local address 127.0.97.62:35843)
0619 04:33:17.030343 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:17.030358 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:17.030401 (+ 43us) server_negotiation.cc:407] Connection header received
0619 04:33:17.032162 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:17.032184 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:17.032626 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:17.032969 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:17.034601 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:17.035887 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:17.037930 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:17.038600 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:17.038735 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:17.042949 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:17.042958 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:17.042973 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:17.043015 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:17.045951 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:17.046567 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:17.046573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:17.046580 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:17.046739 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:17.047096 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:17.047099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:17.047102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:17.047592 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:17.047854 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:17.048327 (+ 473us) server_negotiation.cc:299] Negotiation successful
0619 04:33:17.048747 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":309,"threads_started":1}
I20260619 04:33:17.055796 997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "517ec619de434b00900db7f21b827af1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35843 } }
I20260619 04:33:17.056123 997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:17.056205 997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 517ec619de434b00900db7f21b827af1, State: Initialized, Role: FOLLOWER
I20260619 04:33:17.056746 997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "517ec619de434b00900db7f21b827af1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35843 } }
I20260619 04:33:17.056962 997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:33:17.057072 997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:33:17.057232 997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:17.116535 997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "517ec619de434b00900db7f21b827af1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35843 } }
I20260619 04:33:17.117295 997 leader_election.cc:302] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 517ec619de434b00900db7f21b827af1; no voters:
I20260619 04:33:17.117981 997 leader_election.cc:288] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:33:17.118088 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:17.118651 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1 [term 1 LEADER]: Becoming Leader. State: Replica: 517ec619de434b00900db7f21b827af1, State: Running, Role: LEADER
I20260619 04:33:17.119302 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "517ec619de434b00900db7f21b827af1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35843 } }
I20260619 04:33:17.120983 997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:33:17.122115 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 517ec619de434b00900db7f21b827af1. Latest consensus state: current_term: 1 leader_uuid: "517ec619de434b00900db7f21b827af1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "517ec619de434b00900db7f21b827af1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35843 } } }
I20260619 04:33:17.122148 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "517ec619de434b00900db7f21b827af1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "517ec619de434b00900db7f21b827af1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35843 } } }
I20260619 04:33:17.122643 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1 [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:17.122685 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1 [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:17.125165 1007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:33:17.130582 1007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:33:17.138092 1007 catalog_manager.cc:1347] Generated new cluster ID: e5c9879aec5d4823abaee38854de1e57
I20260619 04:33:17.138195 1007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:33:17.155740 1007 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:33:17.157053 1007 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:33:17.168188 1007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 517ec619de434b00900db7f21b827af1: Generated new TSK 0
I20260619 04:33:17.169008 1007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
W20260619 04:33:18.154847 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.1: real 0.970s user 0.000s sys 0.008s
I20260619 04:33:18.155762 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35843
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:18.454766 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:18.455034 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:18.455181 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:18.462162 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:18.462271 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:18.462416 1023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:33:18.471740 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35843
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:18.472980 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:18.474274 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:18.486157 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:18.486088 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:18.486127 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:18.486790 1023 server_base.cc:1034] running on GCE node
I20260619 04:33:18.487349 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:18.488200 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:18.489389 1023 hybrid_clock.cc:648] HybridClock initialized: now 1781843598489317 us; error 61 us; skew 500 ppm
I20260619 04:33:18.489720 1023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:18 dist-test-slave-n3tq krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:18.492767 1023 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:33:18.494050 1023 webserver.cc:460] Webserver started at http://127.0.97.1:41485/ using document root <none> and password file <none>
I20260619 04:33:18.494680 1023 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:18.494793 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:18.495067 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:18.497867 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "c501d167fcb64a29aa71a8ce490ce99e"
format_stamp: "Formatted at 2026-06-19 04:33:18 on dist-test-slave-n3tq"
server_key: "6e04648c68162b6b668de577f7ec6cc5"
server_key_iv: "0a6fba90f475716f8753a48770c7297c"
server_key_version: "encryptionkey@0"
I20260619 04:33:18.498586 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c501d167fcb64a29aa71a8ce490ce99e"
format_stamp: "Formatted at 2026-06-19 04:33:18 on dist-test-slave-n3tq"
server_key: "6e04648c68162b6b668de577f7ec6cc5"
server_key_iv: "0a6fba90f475716f8753a48770c7297c"
server_key_version: "encryptionkey@0"
I20260619 04:33:18.503394 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260619 04:33:18.506597 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:18.507848 1023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:33:18.507977 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "c501d167fcb64a29aa71a8ce490ce99e"
format_stamp: "Formatted at 2026-06-19 04:33:18 on dist-test-slave-n3tq"
server_key: "6e04648c68162b6b668de577f7ec6cc5"
server_key_iv: "0a6fba90f475716f8753a48770c7297c"
server_key_version: "encryptionkey@0"
I20260619 04:33:18.508136 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:18.537192 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:18.541253 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:18.541503 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:18.542419 1023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:18.543910 1023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:18.543982 1023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:18.544102 1023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:18.544166 1023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:18.572510 1023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41739
I20260619 04:33:18.572561 1150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41739 every 8 connection(s)
I20260619 04:33:18.573767 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:33:18.579430 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 1023
I20260619 04:33:18.579620 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:33:18.580399 388 external_mini_cluster.cc:1428] Setting key 442e4ea6423c01414ca7cf5dddc646ef
Jun 19 04:33:18 dist-test-slave-n3tq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:18.593710 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:18.576126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38035 (local address 127.0.97.62:35843)
0619 04:33:18.576608 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:18.576614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:18.577544 (+ 930us) server_negotiation.cc:407] Connection header received
0619 04:33:18.579395 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:18.579399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:18.579468 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:18.579582 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:18.581446 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:18.582269 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:18.583762 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:18.584149 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:18.584256 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:18.587690 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:18.587696 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:18.587699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:18.587733 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:18.590613 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:18.591317 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:18.591323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:18.591326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:18.591472 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:18.591865 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:18.591870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:18.591874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:18.592240 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:18.592389 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:18.593119 (+ 730us) server_negotiation.cc:299] Negotiation successful
0619 04:33:18.593351 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":206,"threads_started":1}
I20260619 04:33:18.594409 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:18.576733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35843 (local address 127.0.97.1:38035)
0619 04:33:18.577386 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:18.577423 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:18.579092 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:18.579845 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:18.579868 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:18.580594 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:18.581282 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:18.581299 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:18.582418 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:18.582422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:18.583629 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:18.583640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:18.584350 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:18.584358 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:18.584602 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:18.585225 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:18.585246 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:18.587448 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:18.590763 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:18.590773 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:18.590789 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:18.591190 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:18.591564 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:18.591567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:18.591569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:18.591754 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:18.592374 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:18.592383 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:18.592762 (+ 379us) client_negotiation.cc:769] Sending connection context
0619 04:33:18.592981 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 04:33:18.593357 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":255,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 04:33:18.595822 1151 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35843
I20260619 04:33:18.596078 1151 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:18.596784 1151 heartbeater.cc:511] Master 127.0.97.62:35843 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 04:33:18.598829 962 ts_manager.cc:194] Registered new tserver with Master: c501d167fcb64a29aa71a8ce490ce99e (127.0.97.1:41739)
I20260619 04:33:18.600328 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 04:33:18.615639 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35843
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:18.913331 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:18.913584 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:18.913707 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:18.920727 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:18.920835 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:18.920984 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:33:18.930420 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35843
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:18.931629 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:18.932905 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:18.944769 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:18.944701 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:18.944746 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:18.945537 1157 server_base.cc:1034] running on GCE node
I20260619 04:33:18.946041 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:18.946892 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:18.948076 1157 hybrid_clock.cc:648] HybridClock initialized: now 1781843598948031 us; error 40 us; skew 500 ppm
I20260619 04:33:18.948359 1157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:18 dist-test-slave-n3tq krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:18.951201 1157 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:33:18.952612 1157 webserver.cc:460] Webserver started at http://127.0.97.2:42719/ using document root <none> and password file <none>
I20260619 04:33:18.953250 1157 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:18.953361 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:18.953640 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:18.956400 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "8bffe61c0d754408828a7358d5468a34"
format_stamp: "Formatted at 2026-06-19 04:33:18 on dist-test-slave-n3tq"
server_key: "b1a4979233622c839d62b5f3cfd87b68"
server_key_iv: "86285305c062d8eccad859924c62fd95"
server_key_version: "encryptionkey@0"
I20260619 04:33:18.957157 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8bffe61c0d754408828a7358d5468a34"
format_stamp: "Formatted at 2026-06-19 04:33:18 on dist-test-slave-n3tq"
server_key: "b1a4979233622c839d62b5f3cfd87b68"
server_key_iv: "86285305c062d8eccad859924c62fd95"
server_key_version: "encryptionkey@0"
I20260619 04:33:18.962007 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 04:33:18.965269 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:18.966536 1157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:33:18.966671 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "8bffe61c0d754408828a7358d5468a34"
format_stamp: "Formatted at 2026-06-19 04:33:18 on dist-test-slave-n3tq"
server_key: "b1a4979233622c839d62b5f3cfd87b68"
server_key_iv: "86285305c062d8eccad859924c62fd95"
server_key_version: "encryptionkey@0"
I20260619 04:33:18.966840 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:18.979554 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:18.983474 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:18.983722 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:18.984651 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:18.985886 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:18.985958 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:18.986081 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:18.986136 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:19.014683 1157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37983
I20260619 04:33:19.014719 1284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37983 every 8 connection(s)
I20260619 04:33:19.015908 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:33:19.018977 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 1157
I20260619 04:33:19.019126 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:33:19.019548 388 external_mini_cluster.cc:1428] Setting key 9b8ebdb8194806a9b7489fd9e5f25142
Jun 19 04:33:19 dist-test-slave-n3tq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 04:33:19.037647 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:19.018346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33811 (local address 127.0.97.62:35843)
0619 04:33:19.018513 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:19.018518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:19.019707 (+ 1189us) server_negotiation.cc:407] Connection header received
0619 04:33:19.021786 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:19.021791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:19.021876 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:19.021982 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:19.023802 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.024866 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.027153 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.027835 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.027924 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:19.031955 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:19.031960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:19.031964 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:19.031994 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:19.034668 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:19.035450 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:19.035456 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:19.035459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:19.035618 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:19.036010 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:19.036014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:19.036018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:19.036262 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:19.036387 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:19.037107 (+ 720us) server_negotiation.cc:299] Negotiation successful
0619 04:33:19.037322 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:33:19.038676 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:19.018780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35843 (local address 127.0.97.2:33811)
0619 04:33:19.019539 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:19.019589 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:19.021508 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:19.022236 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:19.022247 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:19.022923 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:19.023627 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:19.023647 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.025004 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.025009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:19.027006 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:19.027019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.028204 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.028222 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:19.028524 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:19.029237 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:19.029266 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:19.031737 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:19.034813 (+ 3076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:19.034823 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:19.034841 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:19.035335 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:19.035720 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:19.035725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:19.035728 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:19.035911 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:19.036377 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:19.036384 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:19.036766 (+ 382us) client_negotiation.cc:769] Sending connection context
0619 04:33:19.037006 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 04:33:19.037535 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":308,"threads_started":1}
I20260619 04:33:19.040050 1285 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35843
I20260619 04:33:19.040333 1285 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:19.041038 1285 heartbeater.cc:511] Master 127.0.97.62:35843 requested a full tablet report, sending...
I20260619 04:33:19.042431 962 ts_manager.cc:194] Registered new tserver with Master: 8bffe61c0d754408828a7358d5468a34 (127.0.97.2:37983)
I20260619 04:33:19.043455 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 04:33:19.053608 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35843
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:19.357228 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:19.357517 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:19.357618 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:19.364499 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:19.364600 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:19.364724 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:33:19.374310 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35843
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:19.375567 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:19.376929 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:19.392786 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:19.392742 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:19.392726 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:19.393554 1290 server_base.cc:1034] running on GCE node
I20260619 04:33:19.394109 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:19.395022 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:19.396203 1290 hybrid_clock.cc:648] HybridClock initialized: now 1781843599396168 us; error 49 us; skew 500 ppm
I20260619 04:33:19.396540 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:19 dist-test-slave-n3tq krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:19.399645 1290 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:33:19.401000 1290 webserver.cc:460] Webserver started at http://127.0.97.3:34377/ using document root <none> and password file <none>
I20260619 04:33:19.401638 1290 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:19.401750 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:19.402034 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:19.404927 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "36bdb677f63a4ba4a669dde1784e86bc"
format_stamp: "Formatted at 2026-06-19 04:33:19 on dist-test-slave-n3tq"
server_key: "3fd5950259dfd20dd84138362356b50e"
server_key_iv: "50ef683a15ce3e8678c398c48066f72d"
server_key_version: "encryptionkey@0"
I20260619 04:33:19.405673 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "36bdb677f63a4ba4a669dde1784e86bc"
format_stamp: "Formatted at 2026-06-19 04:33:19 on dist-test-slave-n3tq"
server_key: "3fd5950259dfd20dd84138362356b50e"
server_key_iv: "50ef683a15ce3e8678c398c48066f72d"
server_key_version: "encryptionkey@0"
I20260619 04:33:19.410527 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:33:19.413944 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:19.415225 1290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:33:19.415359 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "36bdb677f63a4ba4a669dde1784e86bc"
format_stamp: "Formatted at 2026-06-19 04:33:19 on dist-test-slave-n3tq"
server_key: "3fd5950259dfd20dd84138362356b50e"
server_key_iv: "50ef683a15ce3e8678c398c48066f72d"
server_key_version: "encryptionkey@0"
I20260619 04:33:19.415524 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:19.443360 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:19.447117 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:19.447381 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:19.448272 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:19.449542 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:19.449616 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:19.449729 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:19.449780 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:19.477448 1290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37025
I20260619 04:33:19.477490 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37025 every 8 connection(s)
I20260619 04:33:19.478627 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:33:19.488823 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 1290
I20260619 04:33:19.488976 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:33:19.489436 388 external_mini_cluster.cc:1428] Setting key 15ffbf2873f5f827f26b121c097c9f24
Jun 19 04:33:19 dist-test-slave-n3tq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:19.497251 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:19.480820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40201 (local address 127.0.97.62:35843)
0619 04:33:19.480978 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:19.480983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:19.482461 (+ 1478us) server_negotiation.cc:407] Connection header received
0619 04:33:19.484193 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:19.484197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:19.484262 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:19.484346 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:19.485735 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.486449 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.487871 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.488240 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.488322 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:19.491796 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:19.491802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:19.491807 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:19.491846 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:19.494336 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:19.495160 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:19.495164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:19.495167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:19.495334 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:19.495755 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:19.495758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:19.495760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:19.496016 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:19.496120 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:19.496840 (+ 720us) server_negotiation.cc:299] Negotiation successful
0619 04:33:19.497003 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 04:33:19.498136 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:19.481271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35843 (local address 127.0.97.3:40201)
0619 04:33:19.482291 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:19.482332 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:19.483942 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:19.484571 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:19.484579 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:19.485064 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:19.485588 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:19.485603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.486556 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.486560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:19.487763 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:19.487773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.488389 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.488394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:19.488623 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:19.489254 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:19.489275 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:19.491584 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:19.494520 (+ 2936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:19.494527 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:19.494540 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:19.495031 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:19.495467 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:19.495471 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:19.495473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:19.495655 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:19.496107 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:19.496112 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:19.496438 (+ 326us) client_negotiation.cc:769] Sending connection context
0619 04:33:19.496717 (+ 279us) client_negotiation.cc:240] Negotiation successful
0619 04:33:19.497093 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"thread_start_us":637,"threads_started":1}
I20260619 04:33:19.499400 1418 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35843
I20260619 04:33:19.499693 1418 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:19.500383 1418 heartbeater.cc:511] Master 127.0.97.62:35843 requested a full tablet report, sending...
I20260619 04:33:19.501698 962 ts_manager.cc:194] Registered new tserver with Master: 36bdb677f63a4ba4a669dde1784e86bc (127.0.97.3:37025)
I20260619 04:33:19.502467 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40201
I20260619 04:33:19.504662 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:33:19.522445 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:19.510788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54856 (local address 127.0.97.62:35843)
0619 04:33:19.510931 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:19.510936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:19.511424 (+ 488us) server_negotiation.cc:407] Connection header received
0619 04:33:19.511547 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:19.511550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:19.511607 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:19.511684 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:19.513144 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.513834 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.515005 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.515430 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.515520 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:19.517523 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:19.517528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:19.517531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:19.517560 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:19.519798 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:19.520869 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:19.520881 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:19.520883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:19.521000 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:19.521285 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:19.521288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:19.521290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:19.521594 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:19.521772 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:19.522028 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 04:33:19.522182 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 04:33:19.528185 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:33:19.534659 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:33:19.557390 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:19.547431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41739 (local address 127.0.0.1:47804)
0619 04:33:19.549301 (+ 1870us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:19.549321 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:19.549429 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:19.550011 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:19.550016 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:19.550037 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:19.550341 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:19.550350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.551546 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.551550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:19.554867 (+ 3317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:19.554877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.556356 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.556361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:19.556575 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:19.556630 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 04:33:19.556752 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 04:33:19.556936 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1724,"mutex_wait_us":579,"thread_start_us":1584,"threads_started":1}
I20260619 04:33:19.557516 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:19.546688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59654 (local address 127.0.97.3:37025)
0619 04:33:19.548235 (+ 1547us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:19.548243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:19.548275 (+ 32us) server_negotiation.cc:407] Connection header received
0619 04:33:19.548384 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:19.548393 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:19.548673 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:19.549549 (+ 876us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:19.549780 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.551381 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.554941 (+ 3560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.556287 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.556781 (+ 494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:19.556982 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:19.557049 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 04:33:19.557191 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1414,"thread_start_us":1264,"threads_started":1}
I20260619 04:33:19.557750 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:19.546022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37025 (local address 127.0.0.1:59654)
0619 04:33:19.547052 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:19.547097 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:19.547262 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:19.548878 (+ 1616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:19.548885 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:19.548918 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:19.549347 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:19.549361 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.551507 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.551512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:19.554781 (+ 3269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:19.554796 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.556522 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.556529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:19.556706 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:19.556828 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 04:33:19.556902 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 04:33:19.557024 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":274,"thread_start_us":635,"threads_started":1}
I20260619 04:33:19.558207 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:19.546569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37983 (local address 127.0.0.1:35248)
0619 04:33:19.548562 (+ 1993us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:19.548578 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:19.548686 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:19.549502 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:19.549506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:19.549529 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:19.549871 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:19.549893 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.551637 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.551640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:19.554364 (+ 2724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:19.554378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.556599 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.556605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:19.556811 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:19.556841 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 04:33:19.556914 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 04:33:19.557017 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1867,"mutex_wait_us":335,"thread_start_us":1747,"threads_started":1}
I20260619 04:33:19.558224 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:19.547896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47804 (local address 127.0.97.1:41739)
0619 04:33:19.548843 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:19.548853 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:19.549544 (+ 691us) server_negotiation.cc:407] Connection header received
0619 04:33:19.549636 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:19.549643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:19.549846 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:19.550011 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:19.550469 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.551391 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.555010 (+ 3619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.556089 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.556214 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:19.556417 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:19.557783 (+ 1366us) server_negotiation.cc:299] Negotiation successful
0619 04:33:19.557964 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":809,"thread_start_us":724,"threads_started":1}
I20260619 04:33:19.558553 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:19.547095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35248 (local address 127.0.97.2:37983)
0619 04:33:19.547804 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:19.547813 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:19.548798 (+ 985us) server_negotiation.cc:407] Connection header received
0619 04:33:19.548901 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:19.548908 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:19.549119 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:19.549295 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:19.550063 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.551517 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.554504 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.556400 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.556528 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:19.556706 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:19.557134 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 04:33:19.557282 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":452,"threads_started":1}
I20260619 04:33:19.561484 1353 tablet_service.cc:1467] Processing CreateTablet for tablet 0733ae3cb2634de6bd037d48a84eae7a (DEFAULT_TABLE table=test-table [id=8ed0dd83244f402a9b306a81621aa28e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:19.561587 1220 tablet_service.cc:1467] Processing CreateTablet for tablet 0733ae3cb2634de6bd037d48a84eae7a (DEFAULT_TABLE table=test-table [id=8ed0dd83244f402a9b306a81621aa28e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:19.561589 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 0733ae3cb2634de6bd037d48a84eae7a (DEFAULT_TABLE table=test-table [id=8ed0dd83244f402a9b306a81621aa28e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:19.562788 1353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0733ae3cb2634de6bd037d48a84eae7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:19.562795 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0733ae3cb2634de6bd037d48a84eae7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:19.563083 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0733ae3cb2634de6bd037d48a84eae7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:19.578351 1438 tablet_bootstrap.cc:492] T 0733ae3cb2634de6bd037d48a84eae7a P c501d167fcb64a29aa71a8ce490ce99e: Bootstrap starting.
I20260619 04:33:19.584408 1438 tablet_bootstrap.cc:654] T 0733ae3cb2634de6bd037d48a84eae7a P c501d167fcb64a29aa71a8ce490ce99e: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:19.585269 1436 tablet_bootstrap.cc:492] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc: Bootstrap starting.
I20260619 04:33:19.589418 1436 tablet_bootstrap.cc:654] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:19.589557 1438 log.cc:824] T 0733ae3cb2634de6bd037d48a84eae7a P c501d167fcb64a29aa71a8ce490ce99e: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:19.590034 1437 tablet_bootstrap.cc:492] T 0733ae3cb2634de6bd037d48a84eae7a P 8bffe61c0d754408828a7358d5468a34: Bootstrap starting.
I20260619 04:33:19.590682 1436 log.cc:824] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:19.593604 1436 tablet_bootstrap.cc:492] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc: No bootstrap required, opened a new log
I20260619 04:33:19.593799 1437 tablet_bootstrap.cc:654] T 0733ae3cb2634de6bd037d48a84eae7a P 8bffe61c0d754408828a7358d5468a34: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:19.593833 1436 ts_tablet_manager.cc:1397] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 04:33:19.595599 1437 log.cc:824] T 0733ae3cb2634de6bd037d48a84eae7a P 8bffe61c0d754408828a7358d5468a34: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:19.598551 1437 tablet_bootstrap.cc:492] T 0733ae3cb2634de6bd037d48a84eae7a P 8bffe61c0d754408828a7358d5468a34: No bootstrap required, opened a new log
I20260619 04:33:19.598553 1438 tablet_bootstrap.cc:492] T 0733ae3cb2634de6bd037d48a84eae7a P c501d167fcb64a29aa71a8ce490ce99e: No bootstrap required, opened a new log
I20260619 04:33:19.598816 1438 ts_tablet_manager.cc:1397] T 0733ae3cb2634de6bd037d48a84eae7a P c501d167fcb64a29aa71a8ce490ce99e: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.012s
I20260619 04:33:19.598814 1437 ts_tablet_manager.cc:1397] T 0733ae3cb2634de6bd037d48a84eae7a P 8bffe61c0d754408828a7358d5468a34: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 04:33:19.601019 1436 raft_consensus.cc:348] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36bdb677f63a4ba4a669dde1784e86bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37025 } } peers { permanent_uuid: "c501d167fcb64a29aa71a8ce490ce99e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41739 } } peers { permanent_uuid: "8bffe61c0d754408828a7358d5468a34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37983 } }
I20260619 04:33:19.601446 1436 raft_consensus.cc:374] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:19.601533 1436 raft_consensus.cc:729] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36bdb677f63a4ba4a669dde1784e86bc, State: Initialized, Role: FOLLOWER
I20260619 04:33:19.602051 1436 consensus_queue.cc:260] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36bdb677f63a4ba4a669dde1784e86bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37025 } } peers { permanent_uuid: "c501d167fcb64a29aa71a8ce490ce99e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41739 } } peers { permanent_uuid: "8bffe61c0d754408828a7358d5468a34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37983 } }
I20260619 04:33:19.603078 1151 heartbeater.cc:503] Master 127.0.97.62:35843 was elected leader, sending a full tablet report...
I20260619 04:33:19.608618 1418 heartbeater.cc:503] Master 127.0.97.62:35843 was elected leader, sending a full tablet report...
I20260619 04:33:19.609108 1436 ts_tablet_manager.cc:1428] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260619 04:33:19.608917 1437 raft_consensus.cc:348] T 0733ae3cb2634de6bd037d48a84eae7a P 8bffe61c0d754408828a7358d5468a34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36bdb677f63a4ba4a669dde1784e86bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37025 } } peers { permanent_uuid: "c501d167fcb64a29aa71a8ce490ce99e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41739 } } peers { permanent_uuid: "8bffe61c0d754408828a7358d5468a34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37983 } }
I20260619 04:33:19.609333 1437 raft_consensus.cc:374] T 0733ae3cb2634de6bd037d48a84eae7a P 8bffe61c0d754408828a7358d5468a34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:19.609429 1437 raft_consensus.cc:729] T 0733ae3cb2634de6bd037d48a84eae7a P 8bffe61c0d754408828a7358d5468a34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bffe61c0d754408828a7358d5468a34, State: Initialized, Role: FOLLOWER
I20260619 04:33:19.610059 1437 consensus_queue.cc:260] T 0733ae3cb2634de6bd037d48a84eae7a P 8bffe61c0d754408828a7358d5468a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36bdb677f63a4ba4a669dde1784e86bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37025 } } peers { permanent_uuid: "c501d167fcb64a29aa71a8ce490ce99e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41739 } } peers { permanent_uuid: "8bffe61c0d754408828a7358d5468a34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37983 } }
I20260619 04:33:19.611526 1285 heartbeater.cc:503] Master 127.0.97.62:35843 was elected leader, sending a full tablet report...
I20260619 04:33:19.611706 1437 ts_tablet_manager.cc:1428] T 0733ae3cb2634de6bd037d48a84eae7a P 8bffe61c0d754408828a7358d5468a34: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260619 04:33:19.610893 1438 raft_consensus.cc:348] T 0733ae3cb2634de6bd037d48a84eae7a P c501d167fcb64a29aa71a8ce490ce99e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36bdb677f63a4ba4a669dde1784e86bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37025 } } peers { permanent_uuid: "c501d167fcb64a29aa71a8ce490ce99e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41739 } } peers { permanent_uuid: "8bffe61c0d754408828a7358d5468a34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37983 } }
I20260619 04:33:19.612097 1438 raft_consensus.cc:374] T 0733ae3cb2634de6bd037d48a84eae7a P c501d167fcb64a29aa71a8ce490ce99e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:19.612193 1438 raft_consensus.cc:729] T 0733ae3cb2634de6bd037d48a84eae7a P c501d167fcb64a29aa71a8ce490ce99e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c501d167fcb64a29aa71a8ce490ce99e, State: Initialized, Role: FOLLOWER
I20260619 04:33:19.612790 1438 consensus_queue.cc:260] T 0733ae3cb2634de6bd037d48a84eae7a P c501d167fcb64a29aa71a8ce490ce99e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36bdb677f63a4ba4a669dde1784e86bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37025 } } peers { permanent_uuid: "c501d167fcb64a29aa71a8ce490ce99e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41739 } } peers { permanent_uuid: "8bffe61c0d754408828a7358d5468a34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37983 } }
I20260619 04:33:19.614904 1438 ts_tablet_manager.cc:1428] T 0733ae3cb2634de6bd037d48a84eae7a P c501d167fcb64a29aa71a8ce490ce99e: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260619 04:33:19.730118 1419 tablet.cc:2369] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:33:19.767764 1286 tablet.cc:2369] T 0733ae3cb2634de6bd037d48a84eae7a P 8bffe61c0d754408828a7358d5468a34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:33:19.794144 1442 raft_consensus.cc:482] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:33:19.794432 1442 raft_consensus.cc:504] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36bdb677f63a4ba4a669dde1784e86bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37025 } } peers { permanent_uuid: "c501d167fcb64a29aa71a8ce490ce99e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41739 } } peers { permanent_uuid: "8bffe61c0d754408828a7358d5468a34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37983 } }
I20260619 04:33:19.796145 1442 leader_election.cc:288] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c501d167fcb64a29aa71a8ce490ce99e (127.0.97.1:41739), 8bffe61c0d754408828a7358d5468a34 (127.0.97.2:37983)
I20260619 04:33:19.802650 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:19.796652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40861 (local address 127.0.97.1:41739)
0619 04:33:19.796850 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:19.796856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:19.796873 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:33:19.796941 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:19.796946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:19.797016 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:19.797095 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:19.797628 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.798661 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.800562 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.801734 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.801857 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:19.801948 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:19.802302 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 04:33:19.802420 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 04:33:19.802650 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:19.796352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41739 (local address 127.0.97.3:40861)
0619 04:33:19.796631 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:19.796650 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:19.796769 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:19.797169 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:19.797178 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:19.797217 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:19.797430 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:19.797445 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.798788 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.798794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:19.800407 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:19.800419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.801895 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.801901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:19.802132 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:19.802161 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 04:33:19.802247 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 04:33:19.802370 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":22}
I20260619 04:33:19.803834 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:19.796815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46765 (local address 127.0.97.2:37983)
0619 04:33:19.797012 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:19.797018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:19.797813 (+ 795us) server_negotiation.cc:407] Connection header received
0619 04:33:19.797954 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:19.797957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:19.798013 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:19.798081 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:19.798572 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.799532 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.801580 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.802613 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.802732 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:19.802829 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:19.803482 (+ 653us) server_negotiation.cc:299] Negotiation successful
0619 04:33:19.803600 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 04:33:19.803709 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0733ae3cb2634de6bd037d48a84eae7a" candidate_uuid: "36bdb677f63a4ba4a669dde1784e86bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c501d167fcb64a29aa71a8ce490ce99e" is_pre_election: true
I20260619 04:33:19.803896 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:19.796999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37983 (local address 127.0.97.3:46765)
0619 04:33:19.797719 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:19.797736 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:19.797847 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:19.798122 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:19.798126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:19.798149 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:19.798443 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:19.798453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.799679 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.799683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:19.801401 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:19.801416 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.803212 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.803217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:19.803327 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:19.803348 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 04:33:19.803436 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 04:33:19.803554 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":1,"thread_start_us":381,"threads_started":1}
I20260619 04:33:19.804232 1106 raft_consensus.cc:2454] T 0733ae3cb2634de6bd037d48a84eae7a P c501d167fcb64a29aa71a8ce490ce99e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36bdb677f63a4ba4a669dde1784e86bc in term 0.
I20260619 04:33:19.804814 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0733ae3cb2634de6bd037d48a84eae7a" candidate_uuid: "36bdb677f63a4ba4a669dde1784e86bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bffe61c0d754408828a7358d5468a34" is_pre_election: true
I20260619 04:33:19.804950 1309 leader_election.cc:302] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36bdb677f63a4ba4a669dde1784e86bc, c501d167fcb64a29aa71a8ce490ce99e; no voters:
I20260619 04:33:19.805335 1442 raft_consensus.cc:2781] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:33:19.805337 1240 raft_consensus.cc:2454] T 0733ae3cb2634de6bd037d48a84eae7a P 8bffe61c0d754408828a7358d5468a34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36bdb677f63a4ba4a669dde1784e86bc in term 0.
I20260619 04:33:19.805469 1442 raft_consensus.cc:482] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:33:19.805555 1442 raft_consensus.cc:3037] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:19.807901 1442 raft_consensus.cc:504] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36bdb677f63a4ba4a669dde1784e86bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37025 } } peers { permanent_uuid: "c501d167fcb64a29aa71a8ce490ce99e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41739 } } peers { permanent_uuid: "8bffe61c0d754408828a7358d5468a34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37983 } }
I20260619 04:33:19.808490 1442 leader_election.cc:288] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc [CANDIDATE]: Term 1 election: Requested vote from peers c501d167fcb64a29aa71a8ce490ce99e (127.0.97.1:41739), 8bffe61c0d754408828a7358d5468a34 (127.0.97.2:37983)
I20260619 04:33:19.808923 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0733ae3cb2634de6bd037d48a84eae7a" candidate_uuid: "36bdb677f63a4ba4a669dde1784e86bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bffe61c0d754408828a7358d5468a34"
I20260619 04:33:19.808936 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0733ae3cb2634de6bd037d48a84eae7a" candidate_uuid: "36bdb677f63a4ba4a669dde1784e86bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c501d167fcb64a29aa71a8ce490ce99e"
I20260619 04:33:19.809087 1240 raft_consensus.cc:3037] T 0733ae3cb2634de6bd037d48a84eae7a P 8bffe61c0d754408828a7358d5468a34 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:19.809108 1106 raft_consensus.cc:3037] T 0733ae3cb2634de6bd037d48a84eae7a P c501d167fcb64a29aa71a8ce490ce99e [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:19.811127 1240 raft_consensus.cc:2454] T 0733ae3cb2634de6bd037d48a84eae7a P 8bffe61c0d754408828a7358d5468a34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36bdb677f63a4ba4a669dde1784e86bc in term 1.
I20260619 04:33:19.811195 1106 raft_consensus.cc:2454] T 0733ae3cb2634de6bd037d48a84eae7a P c501d167fcb64a29aa71a8ce490ce99e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36bdb677f63a4ba4a669dde1784e86bc in term 1.
I20260619 04:33:19.811496 1308 leader_election.cc:302] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36bdb677f63a4ba4a669dde1784e86bc, 8bffe61c0d754408828a7358d5468a34; no voters:
I20260619 04:33:19.811746 1442 raft_consensus.cc:2781] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:19.812141 1442 raft_consensus.cc:686] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc [term 1 LEADER]: Becoming Leader. State: Replica: 36bdb677f63a4ba4a669dde1784e86bc, State: Running, Role: LEADER
I20260619 04:33:19.812646 1442 consensus_queue.cc:237] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36bdb677f63a4ba4a669dde1784e86bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37025 } } peers { permanent_uuid: "c501d167fcb64a29aa71a8ce490ce99e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41739 } } peers { permanent_uuid: "8bffe61c0d754408828a7358d5468a34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37983 } }
I20260619 04:33:19.817188 961 catalog_manager.cc:5495] T 0733ae3cb2634de6bd037d48a84eae7a P 36bdb677f63a4ba4a669dde1784e86bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 36bdb677f63a4ba4a669dde1784e86bc (127.0.97.3). New cstate: current_term: 1 leader_uuid: "36bdb677f63a4ba4a669dde1784e86bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36bdb677f63a4ba4a669dde1784e86bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c501d167fcb64a29aa71a8ce490ce99e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bffe61c0d754408828a7358d5468a34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37983 } health_report { overall_health: UNKNOWN } } }
W20260619 04:33:19.825845 1152 tablet.cc:2369] T 0733ae3cb2634de6bd037d48a84eae7a P c501d167fcb64a29aa71a8ce490ce99e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 04:33:19 dist-test-slave-n3tq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 04:33:19.870430 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:19.856548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47814 (local address 127.0.97.1:41739)
0619 04:33:19.856707 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:19.856713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:19.857129 (+ 416us) server_negotiation.cc:407] Connection header received
0619 04:33:19.857294 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:19.857298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:19.857382 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:19.857463 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:19.858837 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.859553 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.860801 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.861214 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.861320 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:19.865516 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:19.865530 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:19.865545 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:19.865578 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:19.868099 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:19.868781 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:19.868789 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:19.868791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:19.868913 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:19.869211 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:19.869214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:19.869215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:19.869563 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:19.869788 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:19.869998 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 04:33:19.870150 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 04:33:19 dist-test-slave-n3tq krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843599, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 04:33:19 dist-test-slave-n3tq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843599, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 04:33:19.910519 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:19.898225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.0.97.1:41739)
0619 04:33:19.898434 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:19.898440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:19.898640 (+ 200us) server_negotiation.cc:407] Connection header received
0619 04:33:19.898779 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:19.898783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:19.898848 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:19.898943 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:19.900127 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.900906 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.902102 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.902492 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.902582 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:19.906085 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:19.906090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:19.906094 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:19.906123 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:19.908498 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:19.909023 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:19.909026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:19.909028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:19.909145 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:19.909539 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:19.909542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:19.909543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:19.909774 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:19.909972 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:19.910134 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 04:33:19.910289 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 04:33:19 dist-test-slave-n3tq krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 04:33:19 dist-test-slave-n3tq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 04:33:19.951608 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:19.938117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.0.97.1:41739)
0619 04:33:19.938569 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:19.938575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:19.938629 (+ 54us) server_negotiation.cc:407] Connection header received
0619 04:33:19.938842 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:19.938848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:19.938936 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:19.939024 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:19.940325 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.941334 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.942983 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.943633 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.943729 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:19.947437 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:19.947441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:19.947445 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:19.947485 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:19.949676 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:19.950220 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:19.950224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:19.950226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:19.950325 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:19.950578 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:19.950581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:19.950583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:19.950846 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:19.951017 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:19.951198 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 04:33:19.951358 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319}
W20260619 04:33:19.953253 1066 tablet_service.cc:2908] Rejecting scan request for tablet 0733ae3cb2634de6bd037d48a84eae7a: Uninitialized: clean time has not yet been initialized
I20260619 04:33:19.957039 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 1023
I20260619 04:33:19.968019 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:19.957721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47840 (local address 127.0.97.1:41739)
0619 04:33:19.957926 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:19.957932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:19.958152 (+ 220us) server_negotiation.cc:407] Connection header received
0619 04:33:19.958309 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:19.958315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:19.958379 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:19.958453 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:19.959528 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.960180 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.961355 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:19.961728 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:19.961832 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:19.963757 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:19.963763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:19.963766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:19.963793 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:19.965898 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:19.966417 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:19.966422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:19.966424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:19.966526 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:19.966838 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:19.966843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:19.966846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:19.967069 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:19.967334 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:19.967551 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 04:33:19.967737 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:33:19.968649 1146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:20.055394 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 1023
I20260619 04:33:20.069042 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 1157
Jun 19 04:33:20 dist-test-slave-n3tq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 04:33:20.082598 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:20.069741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.0.97.2:37983)
0619 04:33:20.069901 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:20.069906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:20.069924 (+ 18us) server_negotiation.cc:407] Connection header received
0619 04:33:20.070059 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:20.070063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:20.070152 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:20.070238 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:20.071321 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:20.072047 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:20.073238 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:20.073630 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:20.073725 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:20.077369 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:20.077384 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:20.077400 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:20.077435 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:20.080073 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:20.080811 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:20.080820 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:20.080823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:20.080952 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:20.081260 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:20.081263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:20.081265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:20.081614 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:20.081820 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:20.082182 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 04:33:20.082351 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 04:33:20.083214 1280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:20.167985 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 1157
I20260619 04:33:20.180320 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 1290
Jun 19 04:33:20 dist-test-slave-n3tq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 04:33:20.194262 1461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:20.181011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.0.97.3:37025)
0619 04:33:20.181414 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:20.181419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:20.181434 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:33:20.181487 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:20.181491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:20.181554 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:20.181641 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:20.182784 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:20.183447 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:20.184812 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:20.185246 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:20.185337 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:20.189004 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:20.189019 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:20.189036 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:20.189082 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:20.191726 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:20.192337 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:20.192344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:20.192347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:20.192513 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:20.192875 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:20.192880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:20.192884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:20.193204 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:20.193478 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:20.193714 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 04:33:20.193923 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":202,"threads_started":1}
I20260619 04:33:20.194900 1413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:20.280723 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 1290
I20260619 04:33:20.293604 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 930
I20260619 04:33:20.294704 992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:20.398509 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 930
2026-06-19T04:33:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (7870 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:33:20 dist-test-slave-n3tq krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:33:20 dist-test-slave-n3tq krb5kdc[1466](info): set up 2 sockets
Jun 19 04:33:20 dist-test-slave-n3tq krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260619 04:33:21.380266 388 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:33:21 dist-test-slave-n3tq krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:33:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:33:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:33:21.535506 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37807
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:21.837343 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:21.837716 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:21.837868 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:21.845033 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:33:21.845130 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 04:33:21.845194 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:21.845286 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:21.845353 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:33:21.845433 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:33:21.855098 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37807
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:21.856359 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:21.857721 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:21.868497 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:21.868541 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:21.868566 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:21.869176 1479 server_base.cc:1034] running on GCE node
I20260619 04:33:21.869963 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:21.871212 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:21.872431 1479 hybrid_clock.cc:648] HybridClock initialized: now 1781843601872370 us; error 33 us; skew 500 ppm
I20260619 04:33:21.872761 1479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:21 dist-test-slave-n3tq krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:21.875535 1479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:33:21.876860 1479 webserver.cc:460] Webserver started at http://127.0.97.62:32779/ using document root <none> and password file <none>
I20260619 04:33:21.877489 1479 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:21.877579 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:21.877897 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:21.880686 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "7eb1eda4104c425da3e19c36a16f8941"
format_stamp: "Formatted at 2026-06-19 04:33:21 on dist-test-slave-n3tq"
server_key: "e06a38c8b7e750e7207c772acdc3c317"
server_key_iv: "9a36fa95e269b9224e6e60879a82725c"
server_key_version: "encryptionkey@0"
I20260619 04:33:21.881433 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "7eb1eda4104c425da3e19c36a16f8941"
format_stamp: "Formatted at 2026-06-19 04:33:21 on dist-test-slave-n3tq"
server_key: "e06a38c8b7e750e7207c772acdc3c317"
server_key_iv: "9a36fa95e269b9224e6e60879a82725c"
server_key_version: "encryptionkey@0"
I20260619 04:33:21.886092 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 04:33:21.889305 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:21.890522 1479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:33:21.890662 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "7eb1eda4104c425da3e19c36a16f8941"
format_stamp: "Formatted at 2026-06-19 04:33:21 on dist-test-slave-n3tq"
server_key: "e06a38c8b7e750e7207c772acdc3c317"
server_key_iv: "9a36fa95e269b9224e6e60879a82725c"
server_key_version: "encryptionkey@0"
I20260619 04:33:21.890841 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:21.903052 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:21.906514 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:21.906730 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:21.925577 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37807
I20260619 04:33:21.925601 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37807 every 8 connection(s)
I20260619 04:33:21.926848 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:33:21.928861 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 1479
I20260619 04:33:21.929019 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:33:21.929632 388 external_mini_cluster.cc:1428] Setting key ca4012e29dcd7acd0a565d00e7e9e93d
I20260619 04:33:21.930462 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:21.941311 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941: Bootstrap starting.
Jun 19 04:33:21 dist-test-slave-n3tq krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:21.944567 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:21.945526 1556 log.cc:824] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:21.947924 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941: No bootstrap required, opened a new log
I20260619 04:33:21.949610 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:21.931453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42628 (local address 127.0.97.62:37807)
0619 04:33:21.932112 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:21.932124 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:21.932160 (+ 36us) server_negotiation.cc:407] Connection header received
0619 04:33:21.933637 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:21.933657 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:21.934027 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:21.934412 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:21.935738 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:21.937078 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:21.938235 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:21.938910 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:21.939081 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:21.943082 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:21.943090 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:21.943103 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:21.943143 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:21.946011 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:21.946625 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:21.946631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:21.946636 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:21.946784 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:21.947123 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:21.947127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:21.947129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:21.947609 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:21.947859 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:21.948287 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 04:33:21.948665 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":288,"threads_started":1}
I20260619 04:33:21.955263 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb1eda4104c425da3e19c36a16f8941" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37807 } }
I20260619 04:33:21.955659 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:21.955744 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb1eda4104c425da3e19c36a16f8941, State: Initialized, Role: FOLLOWER
I20260619 04:33:21.956238 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb1eda4104c425da3e19c36a16f8941" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37807 } }
I20260619 04:33:21.956446 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:33:21.956585 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:33:21.956727 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:21.958778 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb1eda4104c425da3e19c36a16f8941" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37807 } }
I20260619 04:33:21.959291 1556 leader_election.cc:302] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eb1eda4104c425da3e19c36a16f8941; no voters:
I20260619 04:33:21.959802 1556 leader_election.cc:288] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:33:21.959926 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:21.960553 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb1eda4104c425da3e19c36a16f8941, State: Running, Role: LEADER
I20260619 04:33:21.961071 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb1eda4104c425da3e19c36a16f8941" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37807 } }
I20260619 04:33:21.961678 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:33:21.963543 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eb1eda4104c425da3e19c36a16f8941. Latest consensus state: current_term: 1 leader_uuid: "7eb1eda4104c425da3e19c36a16f8941" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb1eda4104c425da3e19c36a16f8941" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37807 } } }
I20260619 04:33:21.963912 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941 [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:21.965409 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eb1eda4104c425da3e19c36a16f8941" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb1eda4104c425da3e19c36a16f8941" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37807 } } }
I20260619 04:33:21.965821 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941 [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:21.966642 1570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:33:21.970566 1570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:33:21.978189 1570 catalog_manager.cc:1347] Generated new cluster ID: 7025063748194c5ea4839951d2e4722b
I20260619 04:33:21.978296 1570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:33:21.991194 1570 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:33:21.992528 1570 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:33:22.011112 1570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941: Generated new TSK 0
I20260619 04:33:22.011946 1570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 04:33:22.045818 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37807
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:22.350800 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:22.351070 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:22.351163 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:22.358282 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:22.358394 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:22.358484 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 04:33:22.358561 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:33:22.368108 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:22.369344 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:22.370590 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:22.382560 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:22.382638 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:22.382608 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:22.383477 1582 server_base.cc:1034] running on GCE node
I20260619 04:33:22.384006 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:22.384922 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:22.386121 1582 hybrid_clock.cc:648] HybridClock initialized: now 1781843602386056 us; error 39 us; skew 500 ppm
I20260619 04:33:22.386385 1582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:22 dist-test-slave-n3tq krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:22.396749 1582 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:33:22.398193 1582 webserver.cc:460] Webserver started at http://127.0.97.1:38691/ using document root <none> and password file <none>
I20260619 04:33:22.398876 1582 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:22.398968 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:22.399272 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:22.402119 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "797415248e8a4013a7c9118451ea343d"
format_stamp: "Formatted at 2026-06-19 04:33:22 on dist-test-slave-n3tq"
server_key: "c4fd08cf32088fbca20f567213a7e089"
server_key_iv: "45d889e4e6ae0b40ad05dde7dd70aa7e"
server_key_version: "encryptionkey@0"
I20260619 04:33:22.402860 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "797415248e8a4013a7c9118451ea343d"
format_stamp: "Formatted at 2026-06-19 04:33:22 on dist-test-slave-n3tq"
server_key: "c4fd08cf32088fbca20f567213a7e089"
server_key_iv: "45d889e4e6ae0b40ad05dde7dd70aa7e"
server_key_version: "encryptionkey@0"
I20260619 04:33:22.407727 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:33:22.410893 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:22.412104 1582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 04:33:22.412240 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "797415248e8a4013a7c9118451ea343d"
format_stamp: "Formatted at 2026-06-19 04:33:22 on dist-test-slave-n3tq"
server_key: "c4fd08cf32088fbca20f567213a7e089"
server_key_iv: "45d889e4e6ae0b40ad05dde7dd70aa7e"
server_key_version: "encryptionkey@0"
I20260619 04:33:22.412441 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:22.425945 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:22.429934 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:22.430169 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:22.432762 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:22.432842 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:22.432992 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:22.433070 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 04:33:22 dist-test-slave-n3tq krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:22.454671 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:22.433121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57999 (local address 127.0.97.62:37807)
0619 04:33:22.433306 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:22.433312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:22.434670 (+ 1358us) server_negotiation.cc:407] Connection header received
0619 04:33:22.437353 (+ 2683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:22.437359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:22.437446 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:22.437549 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:22.439670 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:22.440506 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:22.443022 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:22.443710 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:22.443852 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:22.447954 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:22.447961 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:22.447967 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:22.448013 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:22.451470 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:22.452276 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:22.452283 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:22.452287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:22.452488 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:22.453001 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:22.453007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:22.453011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:22.453288 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:22.453415 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:22.454118 (+ 703us) server_negotiation.cc:299] Negotiation successful
0619 04:33:22.454349 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 04:33:22.455726 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:22.433643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37807 (local address 127.0.97.1:57999)
0619 04:33:22.434489 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:22.434540 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:22.437030 (+ 2490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:22.437899 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:22.437914 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:22.438648 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:22.439479 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:22.439509 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:22.440644 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:22.440653 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:22.442874 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:22.442889 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:22.443919 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:22.443924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:22.444102 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:22.444786 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:22.444807 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:22.447695 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:22.451624 (+ 3929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:22.451632 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:22.451645 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:22.452143 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:22.452603 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:22.452608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:22.452612 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:22.452882 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:22.453431 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:22.453440 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:22.453780 (+ 340us) client_negotiation.cc:769] Sending connection context
0619 04:33:22.454023 (+ 243us) client_negotiation.cc:240] Negotiation successful
0619 04:33:22.454555 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":342,"threads_started":1}
I20260619 04:33:22.477196 1674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:22.463439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37807 (local address 127.0.97.1:38469)
0619 04:33:22.464007 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:22.464028 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:22.464153 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:22.464503 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:22.464508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:22.464983 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:22.465278 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:22.465287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:22.466380 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:22.466387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:22.468101 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:22.468122 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:22.469195 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:22.469203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:22.469394 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:22.470029 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:22.470053 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:22.470909 (+ 856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:22.474398 (+ 3489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:22.474402 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:22.474404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:22.474787 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:22.475185 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:22.475190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:22.475197 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:22.475371 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:22.475902 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:22.475907 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:22.476135 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 04:33:22.476300 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 04:33:22.476649 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":306,"threads_started":1}
I20260619 04:33:22.477623 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:22.463443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38469 (local address 127.0.97.62:37807)
0619 04:33:22.463609 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:22.463616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:22.464100 (+ 484us) server_negotiation.cc:407] Connection header received
0619 04:33:22.464266 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:22.464269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:22.464356 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:22.464471 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:22.465406 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:22.466187 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:22.468260 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:22.468829 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:22.468955 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:22.471129 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:22.471136 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:22.471142 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:22.471182 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:22.474281 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:22.474891 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:22.474896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:22.474908 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:22.475064 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:22.475477 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:22.475483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:22.475486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:22.475804 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:22.475941 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:22.477080 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0619 04:33:22.477295 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 04:33:22.482883 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33373
I20260619 04:33:22.482911 1716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33373 every 8 connection(s)
I20260619 04:33:22.483978 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:33:22.485690 1717 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37807
I20260619 04:33:22.485883 1717 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:22.486465 1717 heartbeater.cc:511] Master 127.0.97.62:37807 requested a full tablet report, sending...
I20260619 04:33:22.488188 1511 ts_manager.cc:194] Registered new tserver with Master: 797415248e8a4013a7c9118451ea343d (127.0.97.1:33373)
I20260619 04:33:22.489675 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57999
I20260619 04:33:22.491005 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 1582
I20260619 04:33:22.491168 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:33:22.491739 388 external_mini_cluster.cc:1428] Setting key eed722e51822a59688257c58398dcaa3
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 04:33:22.524655 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37807
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:22.835857 1721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:22.836138 1721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:22.836241 1721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:22.844377 1721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:22.844532 1721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:22.844637 1721 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 04:33:22.844717 1721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:33:22.854447 1721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:22.855687 1721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:22.857055 1721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:22.870239 1726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:22.870261 1727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:22.870196 1729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:22.873795 1721 server_base.cc:1034] running on GCE node
I20260619 04:33:22.874472 1721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:22.875422 1721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:22.876674 1721 hybrid_clock.cc:648] HybridClock initialized: now 1781843602876599 us; error 40 us; skew 500 ppm
I20260619 04:33:22.877012 1721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:22 dist-test-slave-n3tq krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:22.880115 1721 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:33:22.881556 1721 webserver.cc:460] Webserver started at http://127.0.97.2:34069/ using document root <none> and password file <none>
I20260619 04:33:22.882256 1721 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:22.882349 1721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:22.882620 1721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:22.885500 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "5e4fd4f3a3f846d0966821ca10938a2b"
format_stamp: "Formatted at 2026-06-19 04:33:22 on dist-test-slave-n3tq"
server_key: "8b753f2c29ea1bcbfcf0a6015be17d48"
server_key_iv: "0cbe5e12d0759e6dae9e20f410b29288"
server_key_version: "encryptionkey@0"
I20260619 04:33:22.886246 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5e4fd4f3a3f846d0966821ca10938a2b"
format_stamp: "Formatted at 2026-06-19 04:33:22 on dist-test-slave-n3tq"
server_key: "8b753f2c29ea1bcbfcf0a6015be17d48"
server_key_iv: "0cbe5e12d0759e6dae9e20f410b29288"
server_key_version: "encryptionkey@0"
I20260619 04:33:22.891232 1721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:33:22.894693 1736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:22.896059 1721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:33:22.896200 1721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "5e4fd4f3a3f846d0966821ca10938a2b"
format_stamp: "Formatted at 2026-06-19 04:33:22 on dist-test-slave-n3tq"
server_key: "8b753f2c29ea1bcbfcf0a6015be17d48"
server_key_iv: "0cbe5e12d0759e6dae9e20f410b29288"
server_key_version: "encryptionkey@0"
I20260619 04:33:22.896402 1721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:22.911123 1721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:22.915218 1721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:22.915478 1721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:22.918152 1721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:22.918229 1721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:22.918351 1721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:22.918401 1721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 04:33:22 dist-test-slave-n3tq krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:22.942834 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:22.918740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41105 (local address 127.0.97.62:37807)
0619 04:33:22.918917 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:22.918924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:22.920483 (+ 1559us) server_negotiation.cc:407] Connection header received
0619 04:33:22.922569 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:22.922576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:22.922666 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:22.922739 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:22.925112 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:22.926247 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:22.929086 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:22.929785 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:22.929925 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:22.935340 (+ 5415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:22.935346 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:22.935351 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:22.935393 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:22.938940 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:22.939983 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:22.939991 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:22.939995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:22.940146 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:22.940674 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:22.940680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:22.940684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:22.941047 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:22.941184 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:22.942260 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0619 04:33:22.942508 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 04:33:22.944443 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:22.919404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37807 (local address 127.0.97.2:41105)
0619 04:33:22.920268 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:22.920324 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:22.922211 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:22.923284 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:22.923295 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:22.924048 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:22.924901 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:22.924930 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:22.926443 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:22.926451 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:22.928923 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:22.928937 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:22.930252 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:22.930260 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:22.930619 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:22.931475 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:22.931506 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:22.934984 (+ 3478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:22.939114 (+ 4130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:22.939124 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:22.939142 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:22.939836 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:22.940260 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:22.940265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:22.940268 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:22.940541 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:22.941189 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:22.941199 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:22.941766 (+ 567us) client_negotiation.cc:769] Sending connection context
0619 04:33:22.942120 (+ 354us) client_negotiation.cc:240] Negotiation successful
0619 04:33:22.942735 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":343,"threads_started":1}
I20260619 04:33:22.963891 1822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:22.950485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37807 (local address 127.0.97.2:52555)
0619 04:33:22.950951 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:22.950965 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:22.951083 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:22.951364 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:22.951368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:22.951781 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:22.952048 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:22.952056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:22.953236 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:22.953246 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:22.954908 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:22.954923 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:22.955818 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:22.955825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:22.956019 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:22.956719 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:22.956745 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:22.957888 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:22.961606 (+ 3718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:22.961610 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:22.961613 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:22.961995 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:22.962363 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:22.962366 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:22.962368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:22.962476 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:22.963063 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:22.963066 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:22.963227 (+ 161us) client_negotiation.cc:769] Sending connection context
0619 04:33:22.963337 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 04:33:22.963551 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":242,"threads_started":1}
I20260619 04:33:22.964015 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:22.950395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52555 (local address 127.0.97.62:37807)
0619 04:33:22.950548 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:22.950553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:22.951039 (+ 486us) server_negotiation.cc:407] Connection header received
0619 04:33:22.951186 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:22.951192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:22.951255 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:22.951346 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:22.952170 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:22.953097 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:22.955039 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:22.955492 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:22.955609 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:22.958121 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:22.958128 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:22.958134 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:22.958174 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:22.961491 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:22.962097 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:22.962103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:22.962108 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:22.962262 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:22.962556 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:22.962560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:22.962564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:22.962961 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:22.963110 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:22.963438 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 04:33:22.963673 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 04:33:22.967316 1721 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33087
I20260619 04:33:22.967341 1855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33087 every 8 connection(s)
I20260619 04:33:22.968528 1721 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:33:22.970388 1856 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37807
I20260619 04:33:22.970449 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 1721
I20260619 04:33:22.970585 1856 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:22.970608 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:33:22.971007 388 external_mini_cluster.cc:1428] Setting key a15f150603c031e1d6da8c2b71cb5762
I20260619 04:33:22.971213 1856 heartbeater.cc:511] Master 127.0.97.62:37807 requested a full tablet report, sending...
I20260619 04:33:22.972640 1510 ts_manager.cc:194] Registered new tserver with Master: 5e4fd4f3a3f846d0966821ca10938a2b (127.0.97.2:33087)
I20260619 04:33:22.973754 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41105
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 04:33:23.005729 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37807
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:23.314281 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:23.314558 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:23.314649 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:23.321952 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:23.322068 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:23.322149 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 04:33:23.322227 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:33:23.332186 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:23.333560 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:23.334894 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:23.347765 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:23.347782 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:23.347826 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:23.348564 1860 server_base.cc:1034] running on GCE node
I20260619 04:33:23.349143 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:23.350062 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:23.351287 1860 hybrid_clock.cc:648] HybridClock initialized: now 1781843603351220 us; error 41 us; skew 500 ppm
I20260619 04:33:23.351605 1860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:23 dist-test-slave-n3tq krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:23.354611 1860 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:33:23.355995 1860 webserver.cc:460] Webserver started at http://127.0.97.3:37749/ using document root <none> and password file <none>
I20260619 04:33:23.356719 1860 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:23.356810 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:23.357108 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:23.359944 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "ad900dd1b6444068833421f0a9be6cd0"
format_stamp: "Formatted at 2026-06-19 04:33:23 on dist-test-slave-n3tq"
server_key: "f33fc4bbd2a3cf9e26e4be10908fc714"
server_key_iv: "8e3ede95468e6c6bbcc51b1eee4e901a"
server_key_version: "encryptionkey@0"
I20260619 04:33:23.360770 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ad900dd1b6444068833421f0a9be6cd0"
format_stamp: "Formatted at 2026-06-19 04:33:23 on dist-test-slave-n3tq"
server_key: "f33fc4bbd2a3cf9e26e4be10908fc714"
server_key_iv: "8e3ede95468e6c6bbcc51b1eee4e901a"
server_key_version: "encryptionkey@0"
I20260619 04:33:23.365698 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 04:33:23.368896 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:23.370136 1860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:33:23.370286 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "ad900dd1b6444068833421f0a9be6cd0"
format_stamp: "Formatted at 2026-06-19 04:33:23 on dist-test-slave-n3tq"
server_key: "f33fc4bbd2a3cf9e26e4be10908fc714"
server_key_iv: "8e3ede95468e6c6bbcc51b1eee4e901a"
server_key_version: "encryptionkey@0"
I20260619 04:33:23.370465 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:23.395062 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:23.399075 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:23.399327 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:23.402165 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:23.402246 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:23.402365 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:23.402418 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 04:33:23 dist-test-slave-n3tq krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:23.424757 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.402743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41781 (local address 127.0.97.62:37807)
0619 04:33:23.402962 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:23.402969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:23.404369 (+ 1400us) server_negotiation.cc:407] Connection header received
0619 04:33:23.406389 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:23.406394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:23.406484 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:23.406600 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:23.408847 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.409957 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.411959 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.412670 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.412798 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.418199 (+ 5401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:23.418207 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:23.418211 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:23.418272 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:23.421225 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:23.422285 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:23.422291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:23.422295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:23.422434 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:23.422905 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:23.422908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:23.422910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:23.423161 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:23.423275 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:23.424203 (+ 928us) server_negotiation.cc:299] Negotiation successful
0619 04:33:23.424395 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 04:33:23.426525 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.403363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37807 (local address 127.0.97.3:41781)
0619 04:33:23.404192 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:23.404242 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:23.406049 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:23.406964 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:23.406975 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:23.407717 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:23.408640 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.408676 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.410117 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.410122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.411813 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.411834 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.413092 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.413099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.413378 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.414231 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:23.414263 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:23.417892 (+ 3629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:23.421381 (+ 3489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:23.421395 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:23.421417 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:23.422131 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:23.422533 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:23.422538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:23.422543 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:23.422808 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:23.423266 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:23.423276 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:23.423790 (+ 514us) client_negotiation.cc:769] Sending connection context
0619 04:33:23.424109 (+ 319us) client_negotiation.cc:240] Negotiation successful
0619 04:33:23.424745 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":365,"threads_started":1}
I20260619 04:33:23.445389 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42809
I20260619 04:33:23.445405 1994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42809 every 8 connection(s)
I20260619 04:33:23.446224 1968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.433267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37807 (local address 127.0.97.3:40399)
0619 04:33:23.433826 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:23.433841 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:23.433971 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:23.434254 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:23.434258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:23.434685 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:23.434959 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.434968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.435913 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.435919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.437276 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.437293 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.438032 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.438042 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.438201 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.438829 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:23.438853 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:23.439866 (+ 1013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:23.443532 (+ 3666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:23.443538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:23.443542 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:23.444001 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:23.444400 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:23.444406 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:23.444410 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:23.444592 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:23.445230 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:23.445235 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:23.445450 (+ 215us) client_negotiation.cc:769] Sending connection context
0619 04:33:23.445618 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 04:33:23.445916 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":318,"threads_started":1}
I20260619 04:33:23.446269 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.433195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40399 (local address 127.0.97.62:37807)
0619 04:33:23.433349 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:23.433353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:23.433930 (+ 577us) server_negotiation.cc:407] Connection header received
0619 04:33:23.434083 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:23.434087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:23.434145 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:23.434221 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:23.435098 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.435790 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.437415 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.437854 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.437968 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.440085 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:23.440091 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:23.440095 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:23.440128 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:23.443239 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:23.444147 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:23.444151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:23.444153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:23.444286 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:23.444735 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:23.444740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:23.444743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:23.445028 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:23.445170 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:23.445733 (+ 563us) server_negotiation.cc:299] Negotiation successful
0619 04:33:23.445940 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 04:33:23.446945 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:33:23.448913 1995 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37807
I20260619 04:33:23.449110 1995 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:23.449841 1995 heartbeater.cc:511] Master 127.0.97.62:37807 requested a full tablet report, sending...
I20260619 04:33:23.451119 1510 ts_manager.cc:194] Registered new tserver with Master: ad900dd1b6444068833421f0a9be6cd0 (127.0.97.3:42809)
I20260619 04:33:23.451869 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41781
I20260619 04:33:23.452015 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 1860
I20260619 04:33:23.452152 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:33:23.452693 388 external_mini_cluster.cc:1428] Setting key d915ee91f889e5b40cce943abaa5ed3e
I20260619 04:33:23.455118 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 04:33:23 dist-test-slave-n3tq krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843603, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 04:33:23 dist-test-slave-n3tq krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843603, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:23.492208 1717 heartbeater.cc:503] Master 127.0.97.62:37807 was elected leader, sending a full tablet report...
I20260619 04:33:23.496544 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.482961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42634 (local address 127.0.97.62:37807)
0619 04:33:23.483141 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:23.483146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:23.483344 (+ 198us) server_negotiation.cc:407] Connection header received
0619 04:33:23.483526 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:23.483530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:23.483610 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:23.483692 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:23.485393 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.486147 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.487900 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.488319 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.488497 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.491892 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:23.491898 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:23.491901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:23.491932 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:23.494488 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:23.495077 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:23.495080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:23.495083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:23.495186 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:23.495462 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:23.495465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:23.495467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:23.495713 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:23.495883 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:23.496091 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 04:33:23.496254 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 04:33:23.500031 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:33:23.506594 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:33:23.541961 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.518426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52062 (local address 127.0.97.3:42809)
0619 04:33:23.533731 (+ 15305us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:23.533743 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:23.533781 (+ 38us) server_negotiation.cc:407] Connection header received
0619 04:33:23.533889 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:23.533899 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:23.534216 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:23.534436 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:23.535256 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.536844 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.539649 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.540872 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.540984 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.541222 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:23.541458 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 04:33:23.541609 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15101,"thread_start_us":16232,"threads_started":1}
I20260619 04:33:23.542019 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.517818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42809 (local address 127.0.0.1:52062)
0619 04:33:23.532205 (+ 14387us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:23.532231 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:23.532367 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:23.534459 (+ 2092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:23.534467 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:23.534511 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:23.534913 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.534925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.536993 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.536999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.539484 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.539497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.541044 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.541050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.541252 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.541294 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 04:33:23.541406 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 04:33:23.541592 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14236,"mutex_wait_us":90,"thread_start_us":12071,"threads_started":1}
I20260619 04:33:23.542303 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.517601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33087 (local address 127.0.0.1:35012)
0619 04:33:23.530394 (+ 12793us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:23.530465 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:23.530713 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:23.534916 (+ 4203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:23.534920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:23.534943 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:23.535360 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.535373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.537308 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.537314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.539552 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.539566 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.541445 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.541451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.541606 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.541638 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 04:33:23.541725 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 04:33:23.541846 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12453,"mutex_wait_us":5,"thread_start_us":14867,"threads_started":1}
I20260619 04:33:23.542878 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.517833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35012 (local address 127.0.97.2:33087)
0619 04:33:23.534252 (+ 16419us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:23.534260 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:23.534288 (+ 28us) server_negotiation.cc:407] Connection header received
0619 04:33:23.534377 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:23.534385 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:23.534628 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:23.534806 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:23.535560 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.537133 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.539710 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.541009 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.541119 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.541306 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:23.542443 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0619 04:33:23.542602 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16263,"thread_start_us":17112,"threads_started":1}
I20260619 04:33:23.546011 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1921b4531047ad8de4f370ba71f691 (DEFAULT_TABLE table=test-table [id=29ce3f5bfff048578160a69ab8f182d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:23.546041 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1921b4531047ad8de4f370ba71f691 (DEFAULT_TABLE table=test-table [id=29ce3f5bfff048578160a69ab8f182d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:23.547324 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1921b4531047ad8de4f370ba71f691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:23.547322 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1921b4531047ad8de4f370ba71f691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:23.547745 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.535303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33373 (local address 127.0.0.1:37198)
0619 04:33:23.538828 (+ 3525us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:23.538849 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:23.538966 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:23.539697 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:23.539701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:23.539724 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:23.540109 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.540120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.542948 (+ 2828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.542954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.545086 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.545100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.546884 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.546909 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.547122 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.547153 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 04:33:23.547239 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 04:33:23.547368 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3408,"thread_start_us":3281,"threads_started":1}
I20260619 04:33:23.547746 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.531344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37198 (local address 127.0.97.1:33373)
0619 04:33:23.538501 (+ 7157us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:23.538511 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:23.538954 (+ 443us) server_negotiation.cc:407] Connection header received
0619 04:33:23.539113 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:23.539121 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:23.539330 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:23.539512 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:23.540276 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.542799 (+ 2523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.545253 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.546629 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.546755 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.546937 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:23.547323 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 04:33:23.547482 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7003,"thread_start_us":6870,"threads_started":1}
I20260619 04:33:23.551076 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1921b4531047ad8de4f370ba71f691 (DEFAULT_TABLE table=test-table [id=29ce3f5bfff048578160a69ab8f182d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:23.552434 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1921b4531047ad8de4f370ba71f691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:23.558851 2014 tablet_bootstrap.cc:492] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b: Bootstrap starting.
I20260619 04:33:23.561730 2014 tablet_bootstrap.cc:654] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:23.562983 2014 log.cc:824] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:23.569059 2017 tablet_bootstrap.cc:492] T 6f1921b4531047ad8de4f370ba71f691 P 797415248e8a4013a7c9118451ea343d: Bootstrap starting.
I20260619 04:33:23.569330 2016 tablet_bootstrap.cc:492] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0: Bootstrap starting.
I20260619 04:33:23.572333 2017 tablet_bootstrap.cc:654] T 6f1921b4531047ad8de4f370ba71f691 P 797415248e8a4013a7c9118451ea343d: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:23.573318 2017 log.cc:824] T 6f1921b4531047ad8de4f370ba71f691 P 797415248e8a4013a7c9118451ea343d: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:23.573555 2016 tablet_bootstrap.cc:654] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:23.573926 2014 tablet_bootstrap.cc:492] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b: No bootstrap required, opened a new log
I20260619 04:33:23.574193 2014 ts_tablet_manager.cc:1397] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20260619 04:33:23.574757 2016 log.cc:824] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:23.577630 2016 tablet_bootstrap.cc:492] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0: No bootstrap required, opened a new log
I20260619 04:33:23.577755 2017 tablet_bootstrap.cc:492] T 6f1921b4531047ad8de4f370ba71f691 P 797415248e8a4013a7c9118451ea343d: No bootstrap required, opened a new log
I20260619 04:33:23.577915 2016 ts_tablet_manager.cc:1397] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 04:33:23.577956 2017 ts_tablet_manager.cc:1397] T 6f1921b4531047ad8de4f370ba71f691 P 797415248e8a4013a7c9118451ea343d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 04:33:23.582085 2014 raft_consensus.cc:348] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } } peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } }
I20260619 04:33:23.582526 2014 raft_consensus.cc:374] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:23.582628 2014 raft_consensus.cc:729] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e4fd4f3a3f846d0966821ca10938a2b, State: Initialized, Role: FOLLOWER
I20260619 04:33:23.583194 2014 consensus_queue.cc:260] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } } peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } }
I20260619 04:33:23.584151 1856 heartbeater.cc:503] Master 127.0.97.62:37807 was elected leader, sending a full tablet report...
I20260619 04:33:23.585980 2014 ts_tablet_manager.cc:1428] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s
I20260619 04:33:23.588120 2017 raft_consensus.cc:348] T 6f1921b4531047ad8de4f370ba71f691 P 797415248e8a4013a7c9118451ea343d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } } peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } }
I20260619 04:33:23.588632 2017 raft_consensus.cc:374] T 6f1921b4531047ad8de4f370ba71f691 P 797415248e8a4013a7c9118451ea343d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:23.588415 2016 raft_consensus.cc:348] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } } peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } }
I20260619 04:33:23.588732 2017 raft_consensus.cc:729] T 6f1921b4531047ad8de4f370ba71f691 P 797415248e8a4013a7c9118451ea343d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 797415248e8a4013a7c9118451ea343d, State: Initialized, Role: FOLLOWER
I20260619 04:33:23.588819 2016 raft_consensus.cc:374] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:23.588912 2016 raft_consensus.cc:729] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad900dd1b6444068833421f0a9be6cd0, State: Initialized, Role: FOLLOWER
I20260619 04:33:23.589322 2017 consensus_queue.cc:260] T 6f1921b4531047ad8de4f370ba71f691 P 797415248e8a4013a7c9118451ea343d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } } peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } }
I20260619 04:33:23.589501 2016 consensus_queue.cc:260] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } } peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } }
I20260619 04:33:23.590409 1995 heartbeater.cc:503] Master 127.0.97.62:37807 was elected leader, sending a full tablet report...
I20260619 04:33:23.590982 2017 ts_tablet_manager.cc:1428] T 6f1921b4531047ad8de4f370ba71f691 P 797415248e8a4013a7c9118451ea343d: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 04:33:23.591480 2016 ts_tablet_manager.cc:1428] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260619 04:33:23.655706 2020 raft_consensus.cc:482] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:33:23.656001 2020 raft_consensus.cc:504] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } } peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } }
I20260619 04:33:23.657616 2020 leader_election.cc:288] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad900dd1b6444068833421f0a9be6cd0 (127.0.97.3:42809), 797415248e8a4013a7c9118451ea343d (127.0.97.1:33373)
I20260619 04:33:23.663655 2022 raft_consensus.cc:482] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:33:23.663900 2022 raft_consensus.cc:504] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } } peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } }
I20260619 04:33:23.664486 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.658584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42925 (local address 127.0.97.1:33373)
0619 04:33:23.658759 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:23.658764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:23.659184 (+ 420us) server_negotiation.cc:407] Connection header received
0619 04:33:23.659359 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:23.659362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:23.659430 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:23.659503 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:23.660008 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.660698 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.662396 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.663208 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.663320 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.663414 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:23.664095 (+ 681us) server_negotiation.cc:299] Negotiation successful
0619 04:33:23.664218 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 04:33:23.664713 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.658501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33373 (local address 127.0.97.2:42925)
0619 04:33:23.659079 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:23.659093 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:23.659219 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:23.659532 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:23.659536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:23.659559 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:23.659861 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.659869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.660818 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.660821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.662239 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.662250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.663654 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.663659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.663910 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.663955 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 04:33:23.664035 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 04:33:23.664218 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":90,"thread_start_us":319,"threads_started":1}
I20260619 04:33:23.664736 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.657848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42809 (local address 127.0.97.2:49513)
0619 04:33:23.658427 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:23.658446 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:23.658633 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:23.659129 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:23.659132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:23.659151 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:23.659436 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.659443 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.660368 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.660372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.661813 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.661824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.663635 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.663646 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.663881 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.663951 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 04:33:23.664038 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 04:33:23.664218 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":65,"spinlock_wait_cycles":5888,"thread_start_us":265,"threads_started":1}
I20260619 04:33:23.665830 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.657901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49513 (local address 127.0.97.3:42809)
0619 04:33:23.658063 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:23.658068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:23.658523 (+ 455us) server_negotiation.cc:407] Connection header received
0619 04:33:23.658812 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:23.658818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:23.658895 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:23.658955 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:23.659566 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.660246 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.661968 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.662752 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.662888 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.662999 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:23.665460 (+ 2461us) server_negotiation.cc:299] Negotiation successful
0619 04:33:23.665578 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 04:33:23.666080 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1921b4531047ad8de4f370ba71f691" candidate_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "797415248e8a4013a7c9118451ea343d" is_pre_election: true
I20260619 04:33:23.665786 2022 leader_election.cc:288] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e4fd4f3a3f846d0966821ca10938a2b (127.0.97.2:33087), 797415248e8a4013a7c9118451ea343d (127.0.97.1:33373)
I20260619 04:33:23.666575 1668 raft_consensus.cc:2454] T 6f1921b4531047ad8de4f370ba71f691 P 797415248e8a4013a7c9118451ea343d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e4fd4f3a3f846d0966821ca10938a2b in term 0.
I20260619 04:33:23.667320 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1921b4531047ad8de4f370ba71f691" candidate_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad900dd1b6444068833421f0a9be6cd0" is_pre_election: true
I20260619 04:33:23.667729 1945 raft_consensus.cc:2454] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e4fd4f3a3f846d0966821ca10938a2b in term 0.
I20260619 04:33:23.667860 1740 leader_election.cc:302] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e4fd4f3a3f846d0966821ca10938a2b, 797415248e8a4013a7c9118451ea343d; no voters:
I20260619 04:33:23.668591 2020 raft_consensus.cc:2781] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:33:23.668733 2020 raft_consensus.cc:482] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:33:23.668836 2020 raft_consensus.cc:3037] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:23.671967 2020 raft_consensus.cc:504] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } } peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } }
I20260619 04:33:23.672683 2020 leader_election.cc:288] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b [CANDIDATE]: Term 1 election: Requested vote from peers ad900dd1b6444068833421f0a9be6cd0 (127.0.97.3:42809), 797415248e8a4013a7c9118451ea343d (127.0.97.1:33373)
I20260619 04:33:23.672842 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.667004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57983 (local address 127.0.97.1:33373)
0619 04:33:23.667147 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:23.667152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:23.667231 (+ 79us) server_negotiation.cc:407] Connection header received
0619 04:33:23.667460 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:23.667466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:23.667528 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:23.667605 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:23.668289 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.669155 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.670784 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.671886 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.672003 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.672095 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:23.672439 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 04:33:23.672579 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 04:33:23.673126 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1921b4531047ad8de4f370ba71f691" candidate_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad900dd1b6444068833421f0a9be6cd0"
I20260619 04:33:23.673117 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1921b4531047ad8de4f370ba71f691" candidate_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "797415248e8a4013a7c9118451ea343d"
I20260619 04:33:23.673312 1945 raft_consensus.cc:3037] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:23.673295 1668 raft_consensus.cc:3037] T 6f1921b4531047ad8de4f370ba71f691 P 797415248e8a4013a7c9118451ea343d [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:23.673564 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.666226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33373 (local address 127.0.97.3:57983)
0619 04:33:23.667149 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:23.667165 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:23.667312 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:23.667736 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:23.667741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:23.667762 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:23.668161 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.668172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.669279 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.669284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.670655 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.670666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.672084 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.672091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.672259 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.672288 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 04:33:23.673032 (+ 744us) client_negotiation.cc:240] Negotiation successful
0619 04:33:23.673157 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":43,"thread_start_us":920,"threads_started":1}
I20260619 04:33:23.674270 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1921b4531047ad8de4f370ba71f691" candidate_uuid: "ad900dd1b6444068833421f0a9be6cd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "797415248e8a4013a7c9118451ea343d" is_pre_election: true
I20260619 04:33:23.675870 1668 raft_consensus.cc:2454] T 6f1921b4531047ad8de4f370ba71f691 P 797415248e8a4013a7c9118451ea343d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e4fd4f3a3f846d0966821ca10938a2b in term 1.
I20260619 04:33:23.676008 1945 raft_consensus.cc:2454] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e4fd4f3a3f846d0966821ca10938a2b in term 1.
I20260619 04:33:23.676187 1740 leader_election.cc:302] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e4fd4f3a3f846d0966821ca10938a2b, 797415248e8a4013a7c9118451ea343d; no voters:
I20260619 04:33:23.676468 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.666645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60657 (local address 127.0.97.2:33087)
0619 04:33:23.666809 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:23.666814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:23.666830 (+ 16us) server_negotiation.cc:407] Connection header received
0619 04:33:23.666881 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:23.666885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:23.666945 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:23.667021 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:23.667539 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.668534 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.671000 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.672219 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.672348 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.672446 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:23.672848 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 04:33:23.676207 (+ 3359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 04:33:23.676482 2020 raft_consensus.cc:2781] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:23.676852 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.666167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33087 (local address 127.0.97.3:60657)
0619 04:33:23.666510 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:23.666526 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:23.666618 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:23.667158 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:23.667163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:23.667191 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:23.667405 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.667418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.668691 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.668696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.670723 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.670742 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.672348 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.672354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.672560 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.672588 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 04:33:23.672667 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 04:33:23.676622 (+ 3955us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":26}
I20260619 04:33:23.676992 2020 raft_consensus.cc:686] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b [term 1 LEADER]: Becoming Leader. State: Replica: 5e4fd4f3a3f846d0966821ca10938a2b, State: Running, Role: LEADER
I20260619 04:33:23.677486 2020 consensus_queue.cc:237] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } } peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } }
I20260619 04:33:23.677538 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1921b4531047ad8de4f370ba71f691" candidate_uuid: "ad900dd1b6444068833421f0a9be6cd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" is_pre_election: true
I20260619 04:33:23.678422 1876 leader_election.cc:302] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad900dd1b6444068833421f0a9be6cd0; no voters: 5e4fd4f3a3f846d0966821ca10938a2b, 797415248e8a4013a7c9118451ea343d
I20260619 04:33:23.678751 2022 raft_consensus.cc:2726] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 04:33:23.682106 1510 catalog_manager.cc:5495] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e4fd4f3a3f846d0966821ca10938a2b (127.0.97.2). New cstate: current_term: 1 leader_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } health_report { overall_health: UNKNOWN } } }
W20260619 04:33:23.698567 1996 tablet.cc:2369] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:33:23.720335 1857 tablet.cc:2369] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:33:23.736130 1718 tablet.cc:2369] T 6f1921b4531047ad8de4f370ba71f691 P 797415248e8a4013a7c9118451ea343d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:33:23.738282 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.732881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.0.97.2:33087)
0619 04:33:23.733087 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:23.733093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:23.733192 (+ 99us) server_negotiation.cc:407] Connection header received
0619 04:33:23.733508 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:23.733512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:23.733620 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:23.733738 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:33:23.734096 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.734820 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.736116 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.736605 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.736709 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.737191 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:33:23.737677 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:33:23.737799 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:23.737923 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 04:33:23.737999 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 04:33:23.745611 1668 raft_consensus.cc:1261] T 6f1921b4531047ad8de4f370ba71f691 P 797415248e8a4013a7c9118451ea343d [term 1 FOLLOWER]: Refusing update from remote peer 5e4fd4f3a3f846d0966821ca10938a2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:33:23.745659 1945 raft_consensus.cc:1261] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0 [term 1 FOLLOWER]: Refusing update from remote peer 5e4fd4f3a3f846d0966821ca10938a2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:33:23.746378 2026 consensus_queue.cc:1035] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:33:23.746703 2020 consensus_queue.cc:1035] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:33:23.756776 2029 mvcc.cc:205] Tried to move back new op lower bound from 7298431400933384192 to 7298431400666669056. Current Snapshot: MvccSnapshot[applied={T|T < 7298431400933384192}]
I20260619 04:33:23.774905 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:33:23.775148 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:33:23.778499 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941: Sending DeleteTablet for 3 replicas of tablet 6f1921b4531047ad8de4f370ba71f691
I20260619 04:33:23.779543 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1921b4531047ad8de4f370ba71f691 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35012
I20260619 04:33:23.779735 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1921b4531047ad8de4f370ba71f691 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52062
I20260619 04:33:23.779838 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1921b4531047ad8de4f370ba71f691 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37198
I20260619 04:33:23.781744 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:33:23.782439 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:33:23.791388 1924 tablet_service.cc:1467] Processing CreateTablet for tablet db42a690fac94308b7cb2a9c1e3c365b (DEFAULT_TABLE table=test-table [id=d07f5be46f4449e0a7b7163a8301ca24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:23.791803 1785 tablet_service.cc:1467] Processing CreateTablet for tablet db42a690fac94308b7cb2a9c1e3c365b (DEFAULT_TABLE table=test-table [id=d07f5be46f4449e0a7b7163a8301ca24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:23.791992 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db42a690fac94308b7cb2a9c1e3c365b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:23.792398 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db42a690fac94308b7cb2a9c1e3c365b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:23.794023 2040 tablet_replica.cc:330] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0: stopping tablet replica
I20260619 04:33:23.795055 2040 raft_consensus.cc:2229] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:33:23.796109 2040 raft_consensus.cc:2258] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:23.797052 1646 tablet_service.cc:1467] Processing CreateTablet for tablet db42a690fac94308b7cb2a9c1e3c365b (DEFAULT_TABLE table=test-table [id=d07f5be46f4449e0a7b7163a8301ca24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:23.797601 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db42a690fac94308b7cb2a9c1e3c365b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:23.802510 2014 tablet_bootstrap.cc:492] T db42a690fac94308b7cb2a9c1e3c365b P 5e4fd4f3a3f846d0966821ca10938a2b: Bootstrap starting.
I20260619 04:33:23.804309 2014 tablet_bootstrap.cc:654] T db42a690fac94308b7cb2a9c1e3c365b P 5e4fd4f3a3f846d0966821ca10938a2b: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:23.806285 2017 tablet_bootstrap.cc:492] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d: Bootstrap starting.
I20260619 04:33:23.807634 2041 tablet_replica.cc:330] T 6f1921b4531047ad8de4f370ba71f691 P 797415248e8a4013a7c9118451ea343d: stopping tablet replica
I20260619 04:33:23.807906 2017 tablet_bootstrap.cc:654] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:23.808135 2041 raft_consensus.cc:2229] T 6f1921b4531047ad8de4f370ba71f691 P 797415248e8a4013a7c9118451ea343d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:33:23.808873 2041 raft_consensus.cc:2258] T 6f1921b4531047ad8de4f370ba71f691 P 797415248e8a4013a7c9118451ea343d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:23.810555 2016 tablet_bootstrap.cc:492] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0: Bootstrap starting.
I20260619 04:33:23.812101 2016 tablet_bootstrap.cc:654] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:23.812145 2041 ts_tablet_manager.cc:1905] T 6f1921b4531047ad8de4f370ba71f691 P 797415248e8a4013a7c9118451ea343d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:23.812251 2040 ts_tablet_manager.cc:1905] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:23.819320 2040 ts_tablet_manager.cc:1918] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:33:23.819510 2040 log.cc:1196] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/6f1921b4531047ad8de4f370ba71f691
I20260619 04:33:23.819994 2040 ts_tablet_manager.cc:1939] T 6f1921b4531047ad8de4f370ba71f691 P ad900dd1b6444068833421f0a9be6cd0: Deleting consensus metadata
I20260619 04:33:23.821868 1498 catalog_manager.cc:4841] TS ad900dd1b6444068833421f0a9be6cd0 (127.0.97.3:42809): tablet 6f1921b4531047ad8de4f370ba71f691 (table test-table [id=29ce3f5bfff048578160a69ab8f182d6]) successfully deleted
I20260619 04:33:23.822850 2039 tablet_replica.cc:330] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b: stopping tablet replica
I20260619 04:33:23.824577 2016 tablet_bootstrap.cc:492] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0: No bootstrap required, opened a new log
I20260619 04:33:23.824719 2016 ts_tablet_manager.cc:1397] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.008s
I20260619 04:33:23.824839 2014 tablet_bootstrap.cc:492] T db42a690fac94308b7cb2a9c1e3c365b P 5e4fd4f3a3f846d0966821ca10938a2b: No bootstrap required, opened a new log
I20260619 04:33:23.825052 2014 ts_tablet_manager.cc:1397] T db42a690fac94308b7cb2a9c1e3c365b P 5e4fd4f3a3f846d0966821ca10938a2b: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.004s
I20260619 04:33:23.825340 2017 tablet_bootstrap.cc:492] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d: No bootstrap required, opened a new log
I20260619 04:33:23.825516 2017 ts_tablet_manager.cc:1397] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.000s
I20260619 04:33:23.825538 2016 raft_consensus.cc:348] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } }
I20260619 04:33:23.825765 2016 raft_consensus.cc:374] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:23.825860 2016 raft_consensus.cc:729] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad900dd1b6444068833421f0a9be6cd0, State: Initialized, Role: FOLLOWER
I20260619 04:33:23.826174 2016 consensus_queue.cc:260] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } }
I20260619 04:33:23.826301 2017 raft_consensus.cc:348] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } }
I20260619 04:33:23.826519 2017 raft_consensus.cc:374] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:23.826608 2017 raft_consensus.cc:729] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 797415248e8a4013a7c9118451ea343d, State: Initialized, Role: FOLLOWER
I20260619 04:33:23.826471 2014 raft_consensus.cc:348] T db42a690fac94308b7cb2a9c1e3c365b P 5e4fd4f3a3f846d0966821ca10938a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } }
I20260619 04:33:23.826859 2016 ts_tablet_manager.cc:1428] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 04:33:23.826823 2017 consensus_queue.cc:260] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } }
I20260619 04:33:23.826910 2014 raft_consensus.cc:374] T db42a690fac94308b7cb2a9c1e3c365b P 5e4fd4f3a3f846d0966821ca10938a2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:23.827123 2014 raft_consensus.cc:729] T db42a690fac94308b7cb2a9c1e3c365b P 5e4fd4f3a3f846d0966821ca10938a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e4fd4f3a3f846d0966821ca10938a2b, State: Initialized, Role: FOLLOWER
I20260619 04:33:23.827323 2017 ts_tablet_manager.cc:1428] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 04:33:23.827436 2014 consensus_queue.cc:260] T db42a690fac94308b7cb2a9c1e3c365b P 5e4fd4f3a3f846d0966821ca10938a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } }
I20260619 04:33:23.828209 2014 ts_tablet_manager.cc:1428] T db42a690fac94308b7cb2a9c1e3c365b P 5e4fd4f3a3f846d0966821ca10938a2b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260619 04:33:23.826211 2039 raft_consensus.cc:2229] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:33:23.828841 2041 ts_tablet_manager.cc:1918] T 6f1921b4531047ad8de4f370ba71f691 P 797415248e8a4013a7c9118451ea343d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:33:23.828976 2041 log.cc:1196] T 6f1921b4531047ad8de4f370ba71f691 P 797415248e8a4013a7c9118451ea343d: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-0/wal/wals/6f1921b4531047ad8de4f370ba71f691
I20260619 04:33:23.829157 2039 raft_consensus.cc:2258] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:23.829494 2041 ts_tablet_manager.cc:1939] T 6f1921b4531047ad8de4f370ba71f691 P 797415248e8a4013a7c9118451ea343d: Deleting consensus metadata
I20260619 04:33:23.831154 1498 catalog_manager.cc:4841] TS 797415248e8a4013a7c9118451ea343d (127.0.97.1:33373): tablet 6f1921b4531047ad8de4f370ba71f691 (table test-table [id=29ce3f5bfff048578160a69ab8f182d6]) successfully deleted
I20260619 04:33:23.834309 2039 ts_tablet_manager.cc:1905] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:23.839526 2039 ts_tablet_manager.cc:1918] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:33:23.839650 2039 log.cc:1196] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/6f1921b4531047ad8de4f370ba71f691
I20260619 04:33:23.840054 2039 ts_tablet_manager.cc:1939] T 6f1921b4531047ad8de4f370ba71f691 P 5e4fd4f3a3f846d0966821ca10938a2b: Deleting consensus metadata
I20260619 04:33:23.841454 1495 catalog_manager.cc:4841] TS 5e4fd4f3a3f846d0966821ca10938a2b (127.0.97.2:33087): tablet 6f1921b4531047ad8de4f370ba71f691 (table test-table [id=29ce3f5bfff048578160a69ab8f182d6]) successfully deleted
I20260619 04:33:23.902293 2021 raft_consensus.cc:482] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:33:23.902537 2021 raft_consensus.cc:504] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } }
I20260619 04:33:23.904609 2021 leader_election.cc:288] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e4fd4f3a3f846d0966821ca10938a2b (127.0.97.2:33087), ad900dd1b6444068833421f0a9be6cd0 (127.0.97.3:42809)
I20260619 04:33:23.912108 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.904973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43481 (local address 127.0.97.2:33087)
0619 04:33:23.905151 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:23.905158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:23.905875 (+ 717us) server_negotiation.cc:407] Connection header received
0619 04:33:23.906085 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:23.906092 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:23.906191 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:23.906252 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:23.907452 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.908191 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.909868 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.911084 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.911209 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.911308 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:23.911795 (+ 487us) server_negotiation.cc:299] Negotiation successful
0619 04:33:23.911905 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:33:23.912113 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.905032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37189 (local address 127.0.97.3:42809)
0619 04:33:23.905293 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:23.905299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:23.905599 (+ 300us) server_negotiation.cc:407] Connection header received
0619 04:33:23.905835 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:23.905841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:23.905917 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:23.905993 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:23.906441 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.907501 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.909997 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.911093 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.911209 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.911308 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:23.911683 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 04:33:23.911820 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260619 04:33:23.912855 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.904804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33087 (local address 127.0.97.1:43481)
0619 04:33:23.905765 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:23.905782 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:23.905921 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:23.906427 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:23.906432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:23.906456 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:23.907260 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.907273 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.908321 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.908326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.909710 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.909721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.911448 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.911452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.911608 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.911637 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 04:33:23.912388 (+ 751us) client_negotiation.cc:240] Negotiation successful
0619 04:33:23.912554 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":180,"thread_start_us":252,"threads_started":1}
I20260619 04:33:23.912961 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.904802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42809 (local address 127.0.97.1:37189)
0619 04:33:23.905494 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:23.905517 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:23.905699 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:23.906028 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:23.906032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:23.906061 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:23.906314 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.906322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.907628 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.907633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.909845 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.909860 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.911298 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.911305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.911494 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.911529 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 04:33:23.912378 (+ 849us) client_negotiation.cc:240] Negotiation successful
0619 04:33:23.912562 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":4,"thread_start_us":669,"threads_started":1}
I20260619 04:33:23.913527 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db42a690fac94308b7cb2a9c1e3c365b" candidate_uuid: "797415248e8a4013a7c9118451ea343d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" is_pre_election: true
I20260619 04:33:23.913617 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db42a690fac94308b7cb2a9c1e3c365b" candidate_uuid: "797415248e8a4013a7c9118451ea343d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad900dd1b6444068833421f0a9be6cd0" is_pre_election: true
I20260619 04:33:23.913803 1806 raft_consensus.cc:2454] T db42a690fac94308b7cb2a9c1e3c365b P 5e4fd4f3a3f846d0966821ca10938a2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 797415248e8a4013a7c9118451ea343d in term 0.
I20260619 04:33:23.913880 1945 raft_consensus.cc:2454] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 797415248e8a4013a7c9118451ea343d in term 0.
I20260619 04:33:23.914109 2022 raft_consensus.cc:482] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:33:23.914247 1598 leader_election.cc:302] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e4fd4f3a3f846d0966821ca10938a2b, 797415248e8a4013a7c9118451ea343d; no voters:
I20260619 04:33:23.914242 2022 raft_consensus.cc:504] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } }
I20260619 04:33:23.914587 2021 raft_consensus.cc:2781] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:33:23.914700 2021 raft_consensus.cc:482] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:33:23.914788 2021 raft_consensus.cc:3037] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:23.914808 2022 leader_election.cc:288] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 797415248e8a4013a7c9118451ea343d (127.0.97.1:33373), 5e4fd4f3a3f846d0966821ca10938a2b (127.0.97.2:33087)
I20260619 04:33:23.915494 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db42a690fac94308b7cb2a9c1e3c365b" candidate_uuid: "ad900dd1b6444068833421f0a9be6cd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" is_pre_election: true
I20260619 04:33:23.915702 1806 raft_consensus.cc:2454] T db42a690fac94308b7cb2a9c1e3c365b P 5e4fd4f3a3f846d0966821ca10938a2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad900dd1b6444068833421f0a9be6cd0 in term 0.
I20260619 04:33:23.916023 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db42a690fac94308b7cb2a9c1e3c365b" candidate_uuid: "ad900dd1b6444068833421f0a9be6cd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "797415248e8a4013a7c9118451ea343d" is_pre_election: true
I20260619 04:33:23.916411 1876 leader_election.cc:302] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e4fd4f3a3f846d0966821ca10938a2b, ad900dd1b6444068833421f0a9be6cd0; no voters:
I20260619 04:33:23.916679 2022 raft_consensus.cc:2781] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:33:23.916788 2022 raft_consensus.cc:482] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:33:23.916875 2022 raft_consensus.cc:3037] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:23.918251 2021 raft_consensus.cc:504] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } }
I20260619 04:33:23.918879 2021 leader_election.cc:288] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d [CANDIDATE]: Term 1 election: Requested vote from peers 5e4fd4f3a3f846d0966821ca10938a2b (127.0.97.2:33087), ad900dd1b6444068833421f0a9be6cd0 (127.0.97.3:42809)
I20260619 04:33:23.919219 1668 raft_consensus.cc:2379] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ad900dd1b6444068833421f0a9be6cd0 in current term 1: Already voted for candidate 797415248e8a4013a7c9118451ea343d in this term.
I20260619 04:33:23.919543 2022 raft_consensus.cc:504] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } }
I20260619 04:33:23.919956 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db42a690fac94308b7cb2a9c1e3c365b" candidate_uuid: "797415248e8a4013a7c9118451ea343d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad900dd1b6444068833421f0a9be6cd0"
I20260619 04:33:23.920176 2022 leader_election.cc:288] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [CANDIDATE]: Term 1 election: Requested vote from peers 797415248e8a4013a7c9118451ea343d (127.0.97.1:33373), 5e4fd4f3a3f846d0966821ca10938a2b (127.0.97.2:33087)
I20260619 04:33:23.920212 1945 raft_consensus.cc:2379] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 797415248e8a4013a7c9118451ea343d in current term 1: Already voted for candidate ad900dd1b6444068833421f0a9be6cd0 in this term.
I20260619 04:33:23.920562 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db42a690fac94308b7cb2a9c1e3c365b" candidate_uuid: "ad900dd1b6444068833421f0a9be6cd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e4fd4f3a3f846d0966821ca10938a2b"
I20260619 04:33:23.920753 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db42a690fac94308b7cb2a9c1e3c365b" candidate_uuid: "797415248e8a4013a7c9118451ea343d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e4fd4f3a3f846d0966821ca10938a2b"
I20260619 04:33:23.920856 1806 raft_consensus.cc:3037] T db42a690fac94308b7cb2a9c1e3c365b P 5e4fd4f3a3f846d0966821ca10938a2b [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:23.921530 1598 leader_election.cc:302] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 797415248e8a4013a7c9118451ea343d; no voters: 5e4fd4f3a3f846d0966821ca10938a2b, ad900dd1b6444068833421f0a9be6cd0
I20260619 04:33:23.921562 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db42a690fac94308b7cb2a9c1e3c365b" candidate_uuid: "ad900dd1b6444068833421f0a9be6cd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "797415248e8a4013a7c9118451ea343d"
I20260619 04:33:23.921758 1668 raft_consensus.cc:2379] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ad900dd1b6444068833421f0a9be6cd0 in current term 1: Already voted for candidate 797415248e8a4013a7c9118451ea343d in this term.
I20260619 04:33:23.922212 2021 raft_consensus.cc:2726] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 04:33:23.923938 1806 raft_consensus.cc:2454] T db42a690fac94308b7cb2a9c1e3c365b P 5e4fd4f3a3f846d0966821ca10938a2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad900dd1b6444068833421f0a9be6cd0 in term 1.
I20260619 04:33:23.924270 1876 leader_election.cc:302] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e4fd4f3a3f846d0966821ca10938a2b, ad900dd1b6444068833421f0a9be6cd0; no voters: 797415248e8a4013a7c9118451ea343d
I20260619 04:33:23.924597 2022 raft_consensus.cc:2781] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:23.925050 2022 raft_consensus.cc:686] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [term 1 LEADER]: Becoming Leader. State: Replica: ad900dd1b6444068833421f0a9be6cd0, State: Running, Role: LEADER
I20260619 04:33:23.925489 2022 consensus_queue.cc:237] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } }
I20260619 04:33:23.933610 1511 catalog_manager.cc:5495] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad900dd1b6444068833421f0a9be6cd0 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ad900dd1b6444068833421f0a9be6cd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } health_report { overall_health: HEALTHY } } }
Jun 19 04:33:23 dist-test-slave-n3tq krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:23.999698 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.986736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37807 (local address 127.0.0.1:42642)
0619 04:33:23.987323 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:23.987342 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:23.987516 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:23.987797 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:23.987801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:23.988479 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:23.988724 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.988734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.989683 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.989688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.990628 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:23.990639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.991297 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.991303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:23.991464 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.992062 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:23.992079 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:23.994608 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:23.997345 (+ 2737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:23.997357 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:23.997366 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:23.997750 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:23.998069 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:23.998074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:23.998078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:23.998201 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:23.998659 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:23.998667 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:23.998824 (+ 157us) client_negotiation.cc:769] Sending connection context
0619 04:33:23.998982 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 04:33:23.999258 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":449,"threads_started":1}
I20260619 04:33:23.999727 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:23.986745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42642 (local address 127.0.97.62:37807)
0619 04:33:23.986925 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:23.986931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:23.987418 (+ 487us) server_negotiation.cc:407] Connection header received
0619 04:33:23.987629 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:23.987634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:23.987689 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:23.987763 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:23.988851 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.989563 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.990752 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:23.991148 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:23.991249 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:23.994774 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:23.994778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:23.994781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:23.994808 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:23.997225 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:23.997852 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:23.997857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:23.997859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:23.997975 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:23.998289 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:23.998294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:23.998297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:23.998535 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:23.998669 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:23.999061 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 04:33:23.999418 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":131,"server-negotiator.queue_time_us":58}
I20260619 04:33:24.004949 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42642:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 04:33:24.005934 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:33:24.012183 1646 tablet_service.cc:1467] Processing CreateTablet for tablet b05d7dfb90d940f4b2e91c76effcc670 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d76565a0f364bae848c97af57a9618e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 04:33:24.012293 1786 tablet_service.cc:1467] Processing CreateTablet for tablet b05d7dfb90d940f4b2e91c76effcc670 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d76565a0f364bae848c97af57a9618e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 04:33:24.012737 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b05d7dfb90d940f4b2e91c76effcc670. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:24.012871 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b05d7dfb90d940f4b2e91c76effcc670. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:24.012704 1925 tablet_service.cc:1467] Processing CreateTablet for tablet b05d7dfb90d940f4b2e91c76effcc670 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d76565a0f364bae848c97af57a9618e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 04:33:24.013187 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b05d7dfb90d940f4b2e91c76effcc670. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:24.020332 2017 tablet_bootstrap.cc:492] T b05d7dfb90d940f4b2e91c76effcc670 P 797415248e8a4013a7c9118451ea343d: Bootstrap starting.
I20260619 04:33:24.020329 2014 tablet_bootstrap.cc:492] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b: Bootstrap starting.
I20260619 04:33:24.021988 2017 tablet_bootstrap.cc:654] T b05d7dfb90d940f4b2e91c76effcc670 P 797415248e8a4013a7c9118451ea343d: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:24.021984 2014 tablet_bootstrap.cc:654] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:24.023365 2016 tablet_bootstrap.cc:492] T b05d7dfb90d940f4b2e91c76effcc670 P ad900dd1b6444068833421f0a9be6cd0: Bootstrap starting.
I20260619 04:33:24.025182 2016 tablet_bootstrap.cc:654] T b05d7dfb90d940f4b2e91c76effcc670 P ad900dd1b6444068833421f0a9be6cd0: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:24.025277 2014 tablet_bootstrap.cc:492] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b: No bootstrap required, opened a new log
I20260619 04:33:24.025451 2014 ts_tablet_manager.cc:1397] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260619 04:33:24.026208 2014 raft_consensus.cc:348] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } }
I20260619 04:33:24.026461 2014 raft_consensus.cc:374] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:24.026540 2014 raft_consensus.cc:729] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e4fd4f3a3f846d0966821ca10938a2b, State: Initialized, Role: FOLLOWER
I20260619 04:33:24.026854 2017 tablet_bootstrap.cc:492] T b05d7dfb90d940f4b2e91c76effcc670 P 797415248e8a4013a7c9118451ea343d: No bootstrap required, opened a new log
I20260619 04:33:24.026783 2014 consensus_queue.cc:260] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } }
I20260619 04:33:24.027000 2017 ts_tablet_manager.cc:1397] T b05d7dfb90d940f4b2e91c76effcc670 P 797415248e8a4013a7c9118451ea343d: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260619 04:33:24.027215 2034 tablet_replica.cc:439] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } } }
I20260619 04:33:24.027565 2034 tablet_replica.cc:442] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:33:24.027989 2014 ts_tablet_manager.cc:1428] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 04:33:24.028371 2017 raft_consensus.cc:348] T b05d7dfb90d940f4b2e91c76effcc670 P 797415248e8a4013a7c9118451ea343d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } }
I20260619 04:33:24.028615 2017 raft_consensus.cc:374] T b05d7dfb90d940f4b2e91c76effcc670 P 797415248e8a4013a7c9118451ea343d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:24.028678 2017 raft_consensus.cc:729] T b05d7dfb90d940f4b2e91c76effcc670 P 797415248e8a4013a7c9118451ea343d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 797415248e8a4013a7c9118451ea343d, State: Initialized, Role: FOLLOWER
I20260619 04:33:24.028915 2017 consensus_queue.cc:260] T b05d7dfb90d940f4b2e91c76effcc670 P 797415248e8a4013a7c9118451ea343d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } }
I20260619 04:33:24.029413 2016 tablet_bootstrap.cc:492] T b05d7dfb90d940f4b2e91c76effcc670 P ad900dd1b6444068833421f0a9be6cd0: No bootstrap required, opened a new log
I20260619 04:33:24.029453 2017 ts_tablet_manager.cc:1428] T b05d7dfb90d940f4b2e91c76effcc670 P 797415248e8a4013a7c9118451ea343d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 04:33:24.029677 2021 tablet_replica.cc:439] T b05d7dfb90d940f4b2e91c76effcc670 P 797415248e8a4013a7c9118451ea343d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } } }
I20260619 04:33:24.029943 2021 tablet_replica.cc:442] T b05d7dfb90d940f4b2e91c76effcc670 P 797415248e8a4013a7c9118451ea343d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:33:24.030169 2016 ts_tablet_manager.cc:1397] T b05d7dfb90d940f4b2e91c76effcc670 P ad900dd1b6444068833421f0a9be6cd0: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260619 04:33:24.031205 2016 raft_consensus.cc:348] T b05d7dfb90d940f4b2e91c76effcc670 P ad900dd1b6444068833421f0a9be6cd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } }
I20260619 04:33:24.031446 2016 raft_consensus.cc:374] T b05d7dfb90d940f4b2e91c76effcc670 P ad900dd1b6444068833421f0a9be6cd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:24.031661 2016 raft_consensus.cc:729] T b05d7dfb90d940f4b2e91c76effcc670 P ad900dd1b6444068833421f0a9be6cd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad900dd1b6444068833421f0a9be6cd0, State: Initialized, Role: FOLLOWER
I20260619 04:33:24.031929 2016 consensus_queue.cc:260] T b05d7dfb90d940f4b2e91c76effcc670 P ad900dd1b6444068833421f0a9be6cd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } }
I20260619 04:33:24.032303 2022 tablet_replica.cc:439] T b05d7dfb90d940f4b2e91c76effcc670 P ad900dd1b6444068833421f0a9be6cd0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } } }
I20260619 04:33:24.032582 2022 tablet_replica.cc:442] T b05d7dfb90d940f4b2e91c76effcc670 P ad900dd1b6444068833421f0a9be6cd0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:33:24.032964 2016 ts_tablet_manager.cc:1428] T b05d7dfb90d940f4b2e91c76effcc670 P ad900dd1b6444068833421f0a9be6cd0: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260619 04:33:24.242553 2034 raft_consensus.cc:482] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:33:24.242743 2034 raft_consensus.cc:504] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } }
I20260619 04:33:24.243332 2034 leader_election.cc:288] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 797415248e8a4013a7c9118451ea343d (127.0.97.1:33373), ad900dd1b6444068833421f0a9be6cd0 (127.0.97.3:42809)
I20260619 04:33:24.243865 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b05d7dfb90d940f4b2e91c76effcc670" candidate_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "797415248e8a4013a7c9118451ea343d" is_pre_election: true
I20260619 04:33:24.243933 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b05d7dfb90d940f4b2e91c76effcc670" candidate_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad900dd1b6444068833421f0a9be6cd0" is_pre_election: true
I20260619 04:33:24.244062 1668 raft_consensus.cc:2454] T b05d7dfb90d940f4b2e91c76effcc670 P 797415248e8a4013a7c9118451ea343d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e4fd4f3a3f846d0966821ca10938a2b in term 0.
I20260619 04:33:24.244179 1945 raft_consensus.cc:2454] T b05d7dfb90d940f4b2e91c76effcc670 P ad900dd1b6444068833421f0a9be6cd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e4fd4f3a3f846d0966821ca10938a2b in term 0.
I20260619 04:33:24.244486 1740 leader_election.cc:302] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e4fd4f3a3f846d0966821ca10938a2b, 797415248e8a4013a7c9118451ea343d; no voters:
I20260619 04:33:24.244714 2034 raft_consensus.cc:2781] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:33:24.244804 2034 raft_consensus.cc:482] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:33:24.244894 2034 raft_consensus.cc:3037] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:24.246855 2034 raft_consensus.cc:504] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } }
I20260619 04:33:24.247329 2034 leader_election.cc:288] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b [CANDIDATE]: Term 1 election: Requested vote from peers 797415248e8a4013a7c9118451ea343d (127.0.97.1:33373), ad900dd1b6444068833421f0a9be6cd0 (127.0.97.3:42809)
I20260619 04:33:24.247835 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b05d7dfb90d940f4b2e91c76effcc670" candidate_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "797415248e8a4013a7c9118451ea343d"
I20260619 04:33:24.247900 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b05d7dfb90d940f4b2e91c76effcc670" candidate_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad900dd1b6444068833421f0a9be6cd0"
I20260619 04:33:24.248044 1668 raft_consensus.cc:3037] T b05d7dfb90d940f4b2e91c76effcc670 P 797415248e8a4013a7c9118451ea343d [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:24.248068 1945 raft_consensus.cc:3037] T b05d7dfb90d940f4b2e91c76effcc670 P ad900dd1b6444068833421f0a9be6cd0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:24.250831 1945 raft_consensus.cc:2454] T b05d7dfb90d940f4b2e91c76effcc670 P ad900dd1b6444068833421f0a9be6cd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e4fd4f3a3f846d0966821ca10938a2b in term 1.
I20260619 04:33:24.250830 1668 raft_consensus.cc:2454] T b05d7dfb90d940f4b2e91c76effcc670 P 797415248e8a4013a7c9118451ea343d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e4fd4f3a3f846d0966821ca10938a2b in term 1.
I20260619 04:33:24.251194 1740 leader_election.cc:302] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e4fd4f3a3f846d0966821ca10938a2b, ad900dd1b6444068833421f0a9be6cd0; no voters:
I20260619 04:33:24.251459 2034 raft_consensus.cc:2781] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:24.251614 2034 raft_consensus.cc:686] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b [term 1 LEADER]: Becoming Leader. State: Replica: 5e4fd4f3a3f846d0966821ca10938a2b, State: Running, Role: LEADER
I20260619 04:33:24.251864 2034 consensus_queue.cc:237] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } }
I20260619 04:33:24.253345 2020 tablet_replica.cc:439] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b: TxnStatusTablet state changed. Reason: New leader 5e4fd4f3a3f846d0966821ca10938a2b. Latest consensus state: current_term: 1 leader_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } } }
I20260619 04:33:24.253619 2020 tablet_replica.cc:442] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b: This TxnStatusTablet replica's current role is: LEADER
I20260619 04:33:24.254222 2059 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 04:33:24.254439 2059 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 04:33:24.254606 1510 catalog_manager.cc:5495] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e4fd4f3a3f846d0966821ca10938a2b (127.0.97.2). New cstate: current_term: 1 leader_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } health_report { overall_health: UNKNOWN } } }
I20260619 04:33:24.323166 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:24.318030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33087 (local address 127.0.0.1:35038)
0619 04:33:24.318243 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:24.318263 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:24.318379 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:24.318723 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:24.318726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:24.318735 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:33:24.318877 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:24.318883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.320204 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.320207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:24.321217 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:24.321228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.322175 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.322179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:24.322314 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:24.322377 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:33:24.322784 (+ 407us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:33:24.322795 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 04:33:24.322853 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 04:33:24.322958 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 04:33:24.323241 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:24.318191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.0.97.2:33087)
0619 04:33:24.318379 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:24.318386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:24.318407 (+ 21us) server_negotiation.cc:407] Connection header received
0619 04:33:24.318529 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:24.318534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:24.318616 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:24.318713 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:33:24.319002 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.320101 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.321340 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.322034 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.322171 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:24.322565 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:33:24.322704 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:33:24.322786 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:24.322893 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 04:33:24.322966 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 04:33:24.327275 1945 raft_consensus.cc:1261] T b05d7dfb90d940f4b2e91c76effcc670 P ad900dd1b6444068833421f0a9be6cd0 [term 1 FOLLOWER]: Refusing update from remote peer 5e4fd4f3a3f846d0966821ca10938a2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:33:24.327333 1667 raft_consensus.cc:1261] T b05d7dfb90d940f4b2e91c76effcc670 P 797415248e8a4013a7c9118451ea343d [term 1 FOLLOWER]: Refusing update from remote peer 5e4fd4f3a3f846d0966821ca10938a2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:33:24.327857 2020 consensus_queue.cc:1035] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:33:24.327914 2022 consensus_queue.cc:1035] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:33:24.328342 2034 consensus_queue.cc:1035] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:33:24.347551 2021 tablet_replica.cc:439] T b05d7dfb90d940f4b2e91c76effcc670 P 797415248e8a4013a7c9118451ea343d: TxnStatusTablet state changed. Reason: New leader 5e4fd4f3a3f846d0966821ca10938a2b. Latest consensus state: current_term: 1 leader_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } } }
I20260619 04:33:24.347872 2021 tablet_replica.cc:442] T b05d7dfb90d940f4b2e91c76effcc670 P 797415248e8a4013a7c9118451ea343d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:33:24.350047 2022 tablet_replica.cc:439] T b05d7dfb90d940f4b2e91c76effcc670 P ad900dd1b6444068833421f0a9be6cd0: TxnStatusTablet state changed. Reason: New leader 5e4fd4f3a3f846d0966821ca10938a2b. Latest consensus state: current_term: 1 leader_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } } }
I20260619 04:33:24.350296 2022 tablet_replica.cc:442] T b05d7dfb90d940f4b2e91c76effcc670 P ad900dd1b6444068833421f0a9be6cd0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:33:24.353958 2066 mvcc.cc:205] Tried to move back new op lower bound from 7298431403318005760 to 7298431403017580544. Current Snapshot: MvccSnapshot[applied={T|T < 7298431403318005760}]
I20260619 04:33:24.355911 2065 consensus_queue.cc:1035] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:33:24.357689 2022 tablet_replica.cc:439] T b05d7dfb90d940f4b2e91c76effcc670 P ad900dd1b6444068833421f0a9be6cd0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } } }
I20260619 04:33:24.357898 2022 tablet_replica.cc:442] T b05d7dfb90d940f4b2e91c76effcc670 P ad900dd1b6444068833421f0a9be6cd0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:33:24.362022 2034 tablet_replica.cc:439] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } } }
I20260619 04:33:24.362339 2034 tablet_replica.cc:442] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b: This TxnStatusTablet replica's current role is: LEADER
I20260619 04:33:24.372715 2069 tablet_replica.cc:439] T b05d7dfb90d940f4b2e91c76effcc670 P 797415248e8a4013a7c9118451ea343d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } } }
I20260619 04:33:24.373065 2069 tablet_replica.cc:442] T b05d7dfb90d940f4b2e91c76effcc670 P 797415248e8a4013a7c9118451ea343d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:33:24.377012 2067 mvcc.cc:205] Tried to move back new op lower bound from 7298431403318005760 to 7298431403017580544. Current Snapshot: MvccSnapshot[applied={T|T < 7298431403318005760 or (T in {7298431403318005760})}]
I20260619 04:33:24.362607 2034 tablet_replica.cc:439] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797415248e8a4013a7c9118451ea343d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33373 } } peers { permanent_uuid: "5e4fd4f3a3f846d0966821ca10938a2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33087 } } peers { permanent_uuid: "ad900dd1b6444068833421f0a9be6cd0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42809 } } }
I20260619 04:33:24.377564 2034 tablet_replica.cc:442] T b05d7dfb90d940f4b2e91c76effcc670 P 5e4fd4f3a3f846d0966821ca10938a2b: This TxnStatusTablet replica's current role is: LEADER
I20260619 04:33:24.377995 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:24.369729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52072 (local address 127.0.97.3:42809)
0619 04:33:24.369892 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:24.369899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:24.370004 (+ 105us) server_negotiation.cc:407] Connection header received
0619 04:33:24.370290 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:24.370296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:24.370391 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:24.370511 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:33:24.371194 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.372288 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.375200 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.375835 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.375925 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:24.376567 (+ 642us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:33:24.377294 (+ 727us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:33:24.377402 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:24.377623 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 04:33:24.377709 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 04:33:24.395053 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:24.389367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44923 (local address 127.0.97.2:33087)
0619 04:33:24.389554 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:24.389561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:24.390004 (+ 443us) server_negotiation.cc:407] Connection header received
0619 04:33:24.390171 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:24.390175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:24.390236 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:24.390309 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:33:24.390746 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.391454 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.393362 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.393821 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.393918 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:24.394415 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:33:24.394538 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:33:24.394604 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:24.394746 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 04:33:24.394809 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 04:33:24.395236 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:24.389324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33087 (local address 127.0.97.3:44923)
0619 04:33:24.389899 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:24.389918 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:24.390068 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:24.390337 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:24.390342 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:24.390354 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:33:24.390593 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:24.390604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.391564 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.391569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:24.393237 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:24.393252 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.393959 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.393966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:24.394168 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:24.394251 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:33:24.394645 (+ 394us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:33:24.394655 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 04:33:24.394728 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 04:33:24.394890 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":299,"threads_started":1}
I20260619 04:33:24.432685 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:24.426186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42195 (local address 127.0.97.3:42809)
0619 04:33:24.426359 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:24.426365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:24.426831 (+ 466us) server_negotiation.cc:407] Connection header received
0619 04:33:24.426973 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:24.426979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:24.427067 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:24.427156 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:33:24.427763 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.428900 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.430743 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.431427 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.431560 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:24.431901 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:33:24.432050 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:33:24.432131 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:24.432276 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 04:33:24.432348 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 04:33:24.432850 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:24.426082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42809 (local address 127.0.97.2:42195)
0619 04:33:24.426745 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:24.426761 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:24.426871 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:24.427171 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:24.427174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:24.427183 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:33:24.427495 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:24.427504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.429036 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.429040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:24.430620 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:24.430632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.431577 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.431582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:24.431706 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:24.431760 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:33:24.432146 (+ 386us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:33:24.432171 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 04:33:24.432248 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 04:33:24.432378 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":314,"threads_started":1}
I20260619 04:33:24.489707 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:33:24.489946 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:33:24.493850 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7eb1eda4104c425da3e19c36a16f8941: Sending DeleteTablet for 3 replicas of tablet db42a690fac94308b7cb2a9c1e3c365b
I20260619 04:33:24.494984 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet db42a690fac94308b7cb2a9c1e3c365b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52062
I20260619 04:33:24.495406 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet db42a690fac94308b7cb2a9c1e3c365b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37198
I20260619 04:33:24.497182 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet db42a690fac94308b7cb2a9c1e3c365b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35012
I20260619 04:33:24.510919 2089 tablet_replica.cc:330] T db42a690fac94308b7cb2a9c1e3c365b P 5e4fd4f3a3f846d0966821ca10938a2b: stopping tablet replica
I20260619 04:33:24.511307 2089 raft_consensus.cc:2229] T db42a690fac94308b7cb2a9c1e3c365b P 5e4fd4f3a3f846d0966821ca10938a2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:33:24.511704 2089 raft_consensus.cc:2258] T db42a690fac94308b7cb2a9c1e3c365b P 5e4fd4f3a3f846d0966821ca10938a2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:24.513027 2089 ts_tablet_manager.cc:1905] T db42a690fac94308b7cb2a9c1e3c365b P 5e4fd4f3a3f846d0966821ca10938a2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:24.520259 2089 ts_tablet_manager.cc:1918] T db42a690fac94308b7cb2a9c1e3c365b P 5e4fd4f3a3f846d0966821ca10938a2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:33:24.520530 2089 log.cc:1196] T db42a690fac94308b7cb2a9c1e3c365b P 5e4fd4f3a3f846d0966821ca10938a2b: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/db42a690fac94308b7cb2a9c1e3c365b
I20260619 04:33:24.521092 2089 ts_tablet_manager.cc:1939] T db42a690fac94308b7cb2a9c1e3c365b P 5e4fd4f3a3f846d0966821ca10938a2b: Deleting consensus metadata
I20260619 04:33:24.522274 1495 catalog_manager.cc:4841] TS 5e4fd4f3a3f846d0966821ca10938a2b (127.0.97.2:33087): tablet db42a690fac94308b7cb2a9c1e3c365b (table test-table [id=d07f5be46f4449e0a7b7163a8301ca24]) successfully deleted
I20260619 04:33:24.527132 2087 tablet_replica.cc:330] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d: stopping tablet replica
I20260619 04:33:24.527403 2087 raft_consensus.cc:2229] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:33:24.527688 2087 raft_consensus.cc:2258] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:24.528833 2087 ts_tablet_manager.cc:1905] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:24.529330 2086 tablet_replica.cc:330] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0: stopping tablet replica
I20260619 04:33:24.534742 2086 raft_consensus.cc:2229] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:33:24.535421 2086 raft_consensus.cc:2258] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:24.541539 2086 ts_tablet_manager.cc:1905] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:24.549401 2087 ts_tablet_manager.cc:1918] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
Jun 19 04:33:24 dist-test-slave-n3tq krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843603, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 04:33:24.557057 2086 ts_tablet_manager.cc:1918] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:33:24.557528 2087 log.cc:1196] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-0/wal/wals/db42a690fac94308b7cb2a9c1e3c365b
I20260619 04:33:24.558391 2087 ts_tablet_manager.cc:1939] T db42a690fac94308b7cb2a9c1e3c365b P 797415248e8a4013a7c9118451ea343d: Deleting consensus metadata
I20260619 04:33:24.559679 1498 catalog_manager.cc:4841] TS 797415248e8a4013a7c9118451ea343d (127.0.97.1:33373): tablet db42a690fac94308b7cb2a9c1e3c365b (table test-table [id=d07f5be46f4449e0a7b7163a8301ca24]) successfully deleted
I20260619 04:33:24.561357 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:24.512927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37202 (local address 127.0.97.1:33373)
0619 04:33:24.526801 (+ 13874us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:24.526808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:24.542322 (+ 15514us) server_negotiation.cc:407] Connection header received
0619 04:33:24.542516 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:24.542522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:24.542606 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:24.542724 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:24.544782 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.545831 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.547661 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.548304 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.548414 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:24.553991 (+ 5577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:24.554010 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:24.554032 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:24.554079 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:24.557781 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:24.558705 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:24.558715 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:24.558718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:24.558920 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:24.559370 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:24.559376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:24.559379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:24.559841 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:24.560115 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:24.560732 (+ 617us) server_negotiation.cc:299] Negotiation successful
0619 04:33:24.560959 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13745,"thread_start_us":13656,"threads_started":1}
I20260619 04:33:24.561839 2086 log.cc:1196] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/db42a690fac94308b7cb2a9c1e3c365b
I20260619 04:33:24.562402 2086 ts_tablet_manager.cc:1939] T db42a690fac94308b7cb2a9c1e3c365b P ad900dd1b6444068833421f0a9be6cd0: Deleting consensus metadata
W20260619 04:33:24.562801 1712 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37202
I20260619 04:33:24.564175 1498 catalog_manager.cc:4841] TS ad900dd1b6444068833421f0a9be6cd0 (127.0.97.3:42809): tablet db42a690fac94308b7cb2a9c1e3c365b (table test-table [id=d07f5be46f4449e0a7b7163a8301ca24]) successfully deleted
I20260619 04:33:24.603695 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:24.578924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42646 (local address 127.0.97.62:37807)
0619 04:33:24.587480 (+ 8556us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:24.587487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:24.587508 (+ 21us) server_negotiation.cc:407] Connection header received
0619 04:33:24.587571 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:24.587577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:24.587654 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:24.587771 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:24.589579 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.590708 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.592442 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.593079 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.593202 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:24.596622 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:24.596629 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:24.596634 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:24.596675 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:24.600150 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:24.601060 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:24.601066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:24.601070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:24.601232 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:24.602081 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:24.602087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:24.602090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:24.602476 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:24.602727 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:24.603059 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 04:33:24.603305 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8431,"thread_start_us":8289,"threads_started":1}
W20260619 04:33:24.604257 1510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42646
I20260619 04:33:24.612737 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 1582
I20260619 04:33:24.630043 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:24.614753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37218 (local address 127.0.97.1:33373)
0619 04:33:24.614941 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:24.614948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:24.614968 (+ 20us) server_negotiation.cc:407] Connection header received
0619 04:33:24.615143 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:24.615149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:24.615228 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:24.615316 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:24.617092 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.618139 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.619916 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.620582 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.620680 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:24.623603 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:24.623609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:24.623615 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:24.623655 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:24.626991 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:24.627743 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:24.627748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:24.627752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:24.627884 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:24.628307 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:24.628312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:24.628316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:24.628655 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:24.628869 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:24.629458 (+ 589us) server_negotiation.cc:299] Negotiation successful
0619 04:33:24.629681 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 04:33:24.631150 1712 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37218
W20260619 04:33:24.631726 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 1582: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:33:24.631958 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 1582
I20260619 04:33:24.654464 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 1721
Jun 19 04:33:24 dist-test-slave-n3tq krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843603, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 04:33:24.674270 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:24.655565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35054 (local address 127.0.97.2:33087)
0619 04:33:24.655740 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:24.655746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:24.655764 (+ 18us) server_negotiation.cc:407] Connection header received
0619 04:33:24.655832 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:24.655837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:24.655930 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:24.656050 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:24.657581 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.658780 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.660619 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.661367 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.661503 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:24.666465 (+ 4962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:24.666484 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:24.666506 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:24.666547 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:24.670943 (+ 4396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:24.671736 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:24.671746 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:24.671750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:24.671901 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:24.672292 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:24.672298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:24.672301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:24.672803 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:24.673122 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:24.673423 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 04:33:24.673635 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 04:33:24.674998 1851 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35054
W20260619 04:33:24.675746 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 1721: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:33:24.675899 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 1721
I20260619 04:33:24.695149 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 1860
Jun 19 04:33:24 dist-test-slave-n3tq krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843603, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 04:33:24.710608 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:24.695851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52086 (local address 127.0.97.3:42809)
0619 04:33:24.696016 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:24.696022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:24.696104 (+ 82us) server_negotiation.cc:407] Connection header received
0619 04:33:24.696270 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:24.696275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:24.696359 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:24.696441 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:24.697616 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.698353 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.699518 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:24.699954 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:24.700060 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:24.704289 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:24.704308 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:24.704331 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:24.704380 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:24.707937 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:24.708615 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:24.708625 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:24.708629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:24.708787 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:24.709110 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:24.709115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:24.709119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:24.709540 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:24.709849 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:24.710012 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 04:33:24.710230 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 04:33:24.711400 1990 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52086
W20260619 04:33:24.712042 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 1860: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:33:24.712191 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 1860
I20260619 04:33:24.726677 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 1479
I20260619 04:33:24.727711 1551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:24.838518 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 1479
2026-06-19T04:33:24Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4441 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:33:24 dist-test-slave-n3tq krb5kdc[2099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:33:24 dist-test-slave-n3tq krb5kdc[2099](info): set up 2 sockets
Jun 19 04:33:24 dist-test-slave-n3tq krb5kdc[2099](info): commencing operation
krb5kdc: starting...
W20260619 04:33:25.828240 388 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:33:25 dist-test-slave-n3tq krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:33:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:33:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:33:25.993216 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37573
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:26.295683 2112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:26.295970 2112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:26.296092 2112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:26.303192 2112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:33:26.303300 2112 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 04:33:26.303347 2112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:26.303437 2112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:26.303507 2112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:33:26.303578 2112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:33:26.313318 2112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37573
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:26.314563 2112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:26.315883 2112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:26.326709 2117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:26.326639 2118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:26.326685 2120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:26.327345 2112 server_base.cc:1034] running on GCE node
I20260619 04:33:26.328063 2112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:26.329468 2112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:26.330695 2112 hybrid_clock.cc:648] HybridClock initialized: now 1781843606330632 us; error 40 us; skew 500 ppm
I20260619 04:33:26.331004 2112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:26 dist-test-slave-n3tq krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:26.333896 2112 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:33:26.335266 2112 webserver.cc:460] Webserver started at http://127.0.97.62:38395/ using document root <none> and password file <none>
I20260619 04:33:26.335955 2112 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:26.336051 2112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:26.336324 2112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:26.339254 2112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "cc27e1ee5edb40ae81b161e23a41c482"
format_stamp: "Formatted at 2026-06-19 04:33:26 on dist-test-slave-n3tq"
server_key: "e394d3b43381f6e7b5df35a2af124a32"
server_key_iv: "3008a3983abee0f9b16f3565a3ce90e6"
server_key_version: "encryptionkey@0"
I20260619 04:33:26.339998 2112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "cc27e1ee5edb40ae81b161e23a41c482"
format_stamp: "Formatted at 2026-06-19 04:33:26 on dist-test-slave-n3tq"
server_key: "e394d3b43381f6e7b5df35a2af124a32"
server_key_iv: "3008a3983abee0f9b16f3565a3ce90e6"
server_key_version: "encryptionkey@0"
I20260619 04:33:26.344973 2112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 04:33:26.348270 2127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:26.349555 2112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:33:26.349700 2112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "cc27e1ee5edb40ae81b161e23a41c482"
format_stamp: "Formatted at 2026-06-19 04:33:26 on dist-test-slave-n3tq"
server_key: "e394d3b43381f6e7b5df35a2af124a32"
server_key_iv: "3008a3983abee0f9b16f3565a3ce90e6"
server_key_version: "encryptionkey@0"
I20260619 04:33:26.349888 2112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:26.360622 2112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:26.364327 2112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:26.364580 2112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:26.383183 2112 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37573
I20260619 04:33:26.383203 2188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37573 every 8 connection(s)
I20260619 04:33:26.384428 2112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:33:26.388120 2189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:26.388278 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 2112
I20260619 04:33:26.388520 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:33:26.389314 388 external_mini_cluster.cc:1428] Setting key c9bef99e19abdccd9ff51f8885386018
I20260619 04:33:26.401419 2189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482: Bootstrap starting.
Jun 19 04:33:26 dist-test-slave-n3tq krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:26.405021 2189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:26.406134 2189 log.cc:824] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:26.409261 2189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482: No bootstrap required, opened a new log
I20260619 04:33:26.411681 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:26.391226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57114 (local address 127.0.97.62:37573)
0619 04:33:26.391943 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:26.391954 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:26.392002 (+ 48us) server_negotiation.cc:407] Connection header received
0619 04:33:26.394115 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:26.394137 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:26.394538 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:26.394855 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:26.396546 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:26.397805 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:26.399451 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:26.400072 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:26.400187 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:26.404612 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:26.404620 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:26.404634 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:26.404667 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:26.407794 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:26.408556 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:26.408562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:26.408568 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:26.408736 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:26.409129 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:26.409132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:26.409134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:26.409639 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:26.409849 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:26.410274 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 04:33:26.410692 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":338,"threads_started":1}
I20260619 04:33:26.416900 2189 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc27e1ee5edb40ae81b161e23a41c482" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37573 } }
I20260619 04:33:26.417210 2189 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:26.417306 2189 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc27e1ee5edb40ae81b161e23a41c482, State: Initialized, Role: FOLLOWER
I20260619 04:33:26.417888 2189 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc27e1ee5edb40ae81b161e23a41c482" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37573 } }
I20260619 04:33:26.418118 2189 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:33:26.418249 2189 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:33:26.418419 2189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:26.420382 2189 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc27e1ee5edb40ae81b161e23a41c482" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37573 } }
I20260619 04:33:26.420886 2189 leader_election.cc:302] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc27e1ee5edb40ae81b161e23a41c482; no voters:
I20260619 04:33:26.421473 2189 leader_election.cc:288] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:33:26.421603 2194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:26.422184 2194 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482 [term 1 LEADER]: Becoming Leader. State: Replica: cc27e1ee5edb40ae81b161e23a41c482, State: Running, Role: LEADER
I20260619 04:33:26.422629 2194 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc27e1ee5edb40ae81b161e23a41c482" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37573 } }
I20260619 04:33:26.423267 2189 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:33:26.425057 2196 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc27e1ee5edb40ae81b161e23a41c482. Latest consensus state: current_term: 1 leader_uuid: "cc27e1ee5edb40ae81b161e23a41c482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc27e1ee5edb40ae81b161e23a41c482" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37573 } } }
I20260619 04:33:26.425436 2196 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482 [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:26.425359 2195 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc27e1ee5edb40ae81b161e23a41c482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc27e1ee5edb40ae81b161e23a41c482" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37573 } } }
I20260619 04:33:26.425593 2195 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482 [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:26.426134 2202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:33:26.431032 2202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:33:26.439404 2202 catalog_manager.cc:1347] Generated new cluster ID: dee6558d8c48436abab67a58147e4c82
I20260619 04:33:26.439524 2202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:33:26.458279 2202 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:33:26.459595 2202 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:33:26.475431 2202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482: Generated new TSK 0
I20260619 04:33:26.476263 2202 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 04:33:26.519820 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37573
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:26.852860 2215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:26.853202 2215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:26.853358 2215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:26.860731 2215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:26.860839 2215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:26.860926 2215 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 04:33:26.860983 2215 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 04:33:26.861050 2215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:33:26.870498 2215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:26.871687 2215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:26.872967 2215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:26.884999 2220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:26.884943 2223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:26.885018 2221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:26.885841 2215 server_base.cc:1034] running on GCE node
I20260619 04:33:26.886359 2215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:26.887184 2215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:26.888370 2215 hybrid_clock.cc:648] HybridClock initialized: now 1781843606888326 us; error 43 us; skew 500 ppm
I20260619 04:33:26.888669 2215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:26 dist-test-slave-n3tq krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:26.891510 2215 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:33:26.892872 2215 webserver.cc:460] Webserver started at http://127.0.97.1:33003/ using document root <none> and password file <none>
I20260619 04:33:26.893493 2215 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:26.893602 2215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:26.893873 2215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:26.896603 2215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "8e9f2ed18bd4467ea123dc19c92adcaf"
format_stamp: "Formatted at 2026-06-19 04:33:26 on dist-test-slave-n3tq"
server_key: "7229b5b7750c13be444a3b28b08013a8"
server_key_iv: "373091dc4f8b5d4e7dd2bd17555347e8"
server_key_version: "encryptionkey@0"
I20260619 04:33:26.897322 2215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8e9f2ed18bd4467ea123dc19c92adcaf"
format_stamp: "Formatted at 2026-06-19 04:33:26 on dist-test-slave-n3tq"
server_key: "7229b5b7750c13be444a3b28b08013a8"
server_key_iv: "373091dc4f8b5d4e7dd2bd17555347e8"
server_key_version: "encryptionkey@0"
I20260619 04:33:26.902024 2215 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 04:33:26.905123 2230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:26.906332 2215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 04:33:26.906477 2215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "8e9f2ed18bd4467ea123dc19c92adcaf"
format_stamp: "Formatted at 2026-06-19 04:33:26 on dist-test-slave-n3tq"
server_key: "7229b5b7750c13be444a3b28b08013a8"
server_key_iv: "373091dc4f8b5d4e7dd2bd17555347e8"
server_key_version: "encryptionkey@0"
I20260619 04:33:26.906646 2215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:26.928155 2215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:26.932523 2215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:26.932785 2215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:26.935328 2215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:26.935462 2215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:26.935575 2215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:26.935668 2215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 04:33:26 dist-test-slave-n3tq krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:26.956483 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:26.935800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49697 (local address 127.0.97.62:37573)
0619 04:33:26.938230 (+ 2430us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:26.938237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:26.938259 (+ 22us) server_negotiation.cc:407] Connection header received
0619 04:33:26.939949 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:26.939954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:26.940038 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:26.940130 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:26.942251 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:26.943346 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:26.945547 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:26.945953 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:26.946061 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:26.949630 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:26.949637 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:26.949642 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:26.949683 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:26.952603 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:26.953632 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:26.953636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:26.953638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:26.953755 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:26.954267 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:26.954272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:26.954274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:26.954631 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:26.954769 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:26.955789 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0619 04:33:26.956039 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2312,"thread_start_us":2224,"threads_started":1}
I20260619 04:33:26.957329 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:26.936564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37573 (local address 127.0.97.1:49697)
0619 04:33:26.937518 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:26.937570 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:26.939645 (+ 2075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:26.940413 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:26.940421 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:26.941184 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:26.942045 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:26.942077 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:26.943505 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:26.943511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:26.945394 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:26.945414 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:26.946173 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:26.946180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:26.946373 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:26.947042 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:26.947067 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:26.949388 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:26.952757 (+ 3369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:26.952769 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:26.952788 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:26.953459 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:26.953848 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:26.953853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:26.953857 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:26.954144 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:26.954741 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:26.954748 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:26.955289 (+ 541us) client_negotiation.cc:769] Sending connection context
0619 04:33:26.955647 (+ 358us) client_negotiation.cc:240] Negotiation successful
0619 04:33:26.956103 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":383,"threads_started":1}
I20260619 04:33:26.979784 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:26.966594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41671 (local address 127.0.97.62:37573)
0619 04:33:26.966807 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:26.966814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:26.967288 (+ 474us) server_negotiation.cc:407] Connection header received
0619 04:33:26.967483 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:26.967486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:26.967552 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:26.967645 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:26.968906 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:26.970003 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:26.971833 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:26.972421 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:26.972577 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:26.974652 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:26.974660 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:26.974665 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:26.974707 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:26.977401 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:26.978026 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:26.978030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:26.978032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:26.978154 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:26.978481 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:26.978486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:26.978489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:26.978730 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:26.978850 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:26.979242 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 04:33:26.979451 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 04:33:26.979869 2310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:26.966511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37573 (local address 127.0.97.1:41671)
0619 04:33:26.967169 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:26.967193 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:26.967358 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:26.967704 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:26.967709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:26.968312 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:26.968734 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:26.968752 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:26.970139 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:26.970144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:26.971681 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:26.971699 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:26.972640 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:26.972645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:26.972763 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:26.973388 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:26.973413 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:26.974459 (+ 1046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:26.977527 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:26.977532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:26.977536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:26.977916 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:26.978272 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:26.978275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:26.978278 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:26.978388 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:26.978834 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:26.978839 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:26.979009 (+ 170us) client_negotiation.cc:769] Sending connection context
0619 04:33:26.979124 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 04:33:26.979445 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":322,"threads_started":1}
I20260619 04:33:26.987337 2215 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35787
I20260619 04:33:26.987392 2350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35787 every 8 connection(s)
I20260619 04:33:26.988608 2215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:33:26.990468 2351 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37573
I20260619 04:33:26.990672 2351 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:26.991264 2351 heartbeater.cc:511] Master 127.0.97.62:37573 requested a full tablet report, sending...
I20260619 04:33:26.993106 2144 ts_manager.cc:194] Registered new tserver with Master: 8e9f2ed18bd4467ea123dc19c92adcaf (127.0.97.1:35787)
I20260619 04:33:26.994516 2144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49697
I20260619 04:33:26.998965 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 2215
I20260619 04:33:26.999128 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:33:26.999661 388 external_mini_cluster.cc:1428] Setting key 58039f9d5f2639946e6011029aaa3982
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 04:33:27.033334 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37573
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:27.341375 2355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:27.341640 2355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:27.341766 2355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:27.348954 2355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:27.349071 2355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:27.349201 2355 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 04:33:27.349262 2355 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 04:33:27.349361 2355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:33:27.359064 2355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:27.360350 2355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:27.361675 2355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:27.373680 2360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:27.373646 2361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:27.373700 2363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:27.374466 2355 server_base.cc:1034] running on GCE node
I20260619 04:33:27.374984 2355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:27.375845 2355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:27.377041 2355 hybrid_clock.cc:648] HybridClock initialized: now 1781843607376974 us; error 53 us; skew 500 ppm
I20260619 04:33:27.377346 2355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:27 dist-test-slave-n3tq krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:27.380208 2355 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:33:27.381570 2355 webserver.cc:460] Webserver started at http://127.0.97.2:37889/ using document root <none> and password file <none>
I20260619 04:33:27.382221 2355 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:27.382328 2355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:27.382635 2355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:27.385428 2355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "1dd08036edf54d5293ef71d27be76c4b"
format_stamp: "Formatted at 2026-06-19 04:33:27 on dist-test-slave-n3tq"
server_key: "a603defc680e234929c26a5ec625892a"
server_key_iv: "15172e17b92d7c36d8264835bd838431"
server_key_version: "encryptionkey@0"
I20260619 04:33:27.386157 2355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1dd08036edf54d5293ef71d27be76c4b"
format_stamp: "Formatted at 2026-06-19 04:33:27 on dist-test-slave-n3tq"
server_key: "a603defc680e234929c26a5ec625892a"
server_key_iv: "15172e17b92d7c36d8264835bd838431"
server_key_version: "encryptionkey@0"
I20260619 04:33:27.391158 2355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:33:27.394536 2370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:27.395874 2355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:33:27.396008 2355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "1dd08036edf54d5293ef71d27be76c4b"
format_stamp: "Formatted at 2026-06-19 04:33:27 on dist-test-slave-n3tq"
server_key: "a603defc680e234929c26a5ec625892a"
server_key_iv: "15172e17b92d7c36d8264835bd838431"
server_key_version: "encryptionkey@0"
I20260619 04:33:27.396179 2355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:27.429060 2355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:27.433073 2355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:27.433328 2355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:27.435756 2355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:27.435854 2355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:27.435959 2355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:27.436031 2355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 04:33:27 dist-test-slave-n3tq krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:27.458703 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:27.436416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58919 (local address 127.0.97.62:37573)
0619 04:33:27.436627 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:27.436633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:27.438025 (+ 1392us) server_negotiation.cc:407] Connection header received
0619 04:33:27.440170 (+ 2145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:27.440176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:27.440264 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:27.440339 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:27.442876 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:27.444058 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:27.446210 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:27.446760 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:27.446892 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:27.451613 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:27.451621 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:27.451626 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:27.451674 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:27.455074 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:27.455951 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:27.455957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:27.455971 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:27.456138 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:27.456673 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:27.456678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:27.456682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:27.456999 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:27.457151 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:27.458137 (+ 986us) server_negotiation.cc:299] Negotiation successful
0619 04:33:27.458379 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 04:33:27.459810 2379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:27.437045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37573 (local address 127.0.97.2:58919)
0619 04:33:27.437863 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:27.437916 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:27.439821 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:27.441054 (+ 1233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:27.441061 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:27.441814 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:27.442665 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:27.442695 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:27.444216 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:27.444221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:27.446075 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:27.446093 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:27.447037 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:27.447050 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:27.447370 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:27.448259 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:27.448294 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:27.451277 (+ 2983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:27.455218 (+ 3941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:27.455228 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:27.455250 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:27.455806 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:27.456261 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:27.456266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:27.456270 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:27.456504 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:27.457233 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:27.457245 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:27.457702 (+ 457us) client_negotiation.cc:769] Sending connection context
0619 04:33:27.458016 (+ 314us) client_negotiation.cc:240] Negotiation successful
0619 04:33:27.458559 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":339,"threads_started":1}
I20260619 04:33:27.478322 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:27.464686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46329 (local address 127.0.97.62:37573)
0619 04:33:27.464836 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:27.464842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:27.465812 (+ 970us) server_negotiation.cc:407] Connection header received
0619 04:33:27.465975 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:27.465979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:27.466037 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:27.466105 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:27.467210 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:27.467926 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:27.469992 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:27.470373 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:27.470461 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:27.472822 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:27.472829 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:27.472833 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:27.472870 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:27.475838 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:27.476666 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:27.476670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:27.476672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:27.476781 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:27.477128 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:27.477131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:27.477133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:27.477423 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:27.477528 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:27.477946 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 04:33:27.478107 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 04:33:27.478695 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:27.464959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37573 (local address 127.0.97.2:46329)
0619 04:33:27.465704 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:27.465724 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:27.465868 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:27.466148 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:27.466153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:27.466717 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:27.467090 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:27.467103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:27.468039 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:27.468045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:27.469854 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:27.469872 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:27.470552 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:27.470559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:27.470731 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:27.471453 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:27.471470 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:27.472590 (+ 1120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:27.475970 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:27.475976 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:27.475981 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:27.476556 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:27.476872 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:27.476877 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:27.476881 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:27.477042 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:27.477518 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:27.477522 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:27.477728 (+ 206us) client_negotiation.cc:769] Sending connection context
0619 04:33:27.477867 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 04:33:27.478214 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":429,"threads_started":1}
I20260619 04:33:27.481182 2355 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43405
I20260619 04:33:27.481220 2489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43405 every 8 connection(s)
I20260619 04:33:27.482384 2355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:33:27.484488 2490 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37573
I20260619 04:33:27.484699 2490 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:27.485347 2490 heartbeater.cc:511] Master 127.0.97.62:37573 requested a full tablet report, sending...
I20260619 04:33:27.486534 2144 ts_manager.cc:194] Registered new tserver with Master: 1dd08036edf54d5293ef71d27be76c4b (127.0.97.2:43405)
I20260619 04:33:27.487291 2144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58919
I20260619 04:33:27.489114 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 2355
I20260619 04:33:27.489269 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:33:27.489639 388 external_mini_cluster.cc:1428] Setting key 8c29f4d64224096303e84074ec0fa300
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 04:33:27.522715 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37573
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:27.827294 2494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:27.827577 2494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:27.827706 2494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:27.834941 2494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:27.835055 2494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:27.835165 2494 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 04:33:27.835224 2494 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 04:33:27.835299 2494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:33:27.845005 2494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:27.846266 2494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:27.847589 2494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:27.860601 2499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:27.860626 2502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:27.861114 2494 server_base.cc:1034] running on GCE node
W20260619 04:33:27.860708 2500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:27.861869 2494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:27.862831 2494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:27.864030 2494 hybrid_clock.cc:648] HybridClock initialized: now 1781843607863984 us; error 42 us; skew 500 ppm
I20260619 04:33:27.864318 2494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:27 dist-test-slave-n3tq krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:27.867274 2494 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:33:27.868651 2494 webserver.cc:460] Webserver started at http://127.0.97.3:38757/ using document root <none> and password file <none>
I20260619 04:33:27.869311 2494 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:27.869421 2494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:27.869700 2494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:27.872545 2494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "26e97c9f758a4e46a1afee4c07288303"
format_stamp: "Formatted at 2026-06-19 04:33:27 on dist-test-slave-n3tq"
server_key: "b18ffaaf3d7c88419920ee64a138af9d"
server_key_iv: "7358eb3164d0be90dce66c567b082116"
server_key_version: "encryptionkey@0"
I20260619 04:33:27.873260 2494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "26e97c9f758a4e46a1afee4c07288303"
format_stamp: "Formatted at 2026-06-19 04:33:27 on dist-test-slave-n3tq"
server_key: "b18ffaaf3d7c88419920ee64a138af9d"
server_key_iv: "7358eb3164d0be90dce66c567b082116"
server_key_version: "encryptionkey@0"
I20260619 04:33:27.878103 2494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 04:33:27.881487 2509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:27.882803 2494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:33:27.882942 2494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "26e97c9f758a4e46a1afee4c07288303"
format_stamp: "Formatted at 2026-06-19 04:33:27 on dist-test-slave-n3tq"
server_key: "b18ffaaf3d7c88419920ee64a138af9d"
server_key_iv: "7358eb3164d0be90dce66c567b082116"
server_key_version: "encryptionkey@0"
I20260619 04:33:27.883124 2494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:27.894423 2494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:27.897957 2494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:27.898204 2494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:27.900640 2494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:27.900738 2494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:27.900842 2494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:27.900918 2494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 04:33:27 dist-test-slave-n3tq krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:27.922241 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:27.901202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51573 (local address 127.0.97.62:37573)
0619 04:33:27.901393 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:27.901400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:27.902633 (+ 1233us) server_negotiation.cc:407] Connection header received
0619 04:33:27.904722 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:27.904728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:27.904819 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:27.904936 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:27.906921 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:27.907885 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:27.910670 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:27.911288 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:27.911404 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:27.915439 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:27.915444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:27.915447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:27.915480 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:27.918502 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:27.919402 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:27.919408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:27.919411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:27.919585 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:27.920155 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:27.920160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:27.920164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:27.920433 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:27.920599 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:27.921631 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0619 04:33:27.921872 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 04:33:27.924271 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:27.901679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37573 (local address 127.0.97.3:51573)
0619 04:33:27.902459 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:27.902510 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:27.904387 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:27.905235 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:27.905247 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:27.905918 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:27.906738 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:27.906769 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:27.908051 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:27.908058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:27.910517 (+ 2459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:27.910534 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:27.911510 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:27.911517 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:27.911736 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:27.912499 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:27.912535 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:27.915213 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:27.918638 (+ 3425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:27.918647 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:27.918666 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:27.919260 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:27.919730 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:27.919735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:27.919744 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:27.920027 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:27.920582 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:27.920595 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:27.921116 (+ 521us) client_negotiation.cc:769] Sending connection context
0619 04:33:27.921487 (+ 371us) client_negotiation.cc:240] Negotiation successful
0619 04:33:27.922148 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":337,"threads_started":1}
I20260619 04:33:27.943154 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:27.929746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37573 (local address 127.0.97.3:47895)
0619 04:33:27.930278 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:27.930295 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:27.930419 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:27.930785 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:27.930791 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:27.931437 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:27.931986 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:27.932003 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:27.933000 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:27.933005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:27.934660 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:27.934674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:27.935334 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:27.935341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:27.935519 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:27.936146 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:27.936167 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:27.937467 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:27.940646 (+ 3179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:27.940652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:27.940656 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:27.941212 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:27.941596 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:27.941601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:27.941605 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:27.941741 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:27.942343 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:27.942346 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:27.942491 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 04:33:27.942595 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 04:33:27.942828 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":304,"threads_started":1}
I20260619 04:33:27.943307 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:27.929687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47895 (local address 127.0.97.62:37573)
0619 04:33:27.929835 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:27.929839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:27.930375 (+ 536us) server_negotiation.cc:407] Connection header received
0619 04:33:27.930536 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:27.930541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:27.930633 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:27.930722 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:27.932139 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:27.932893 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:27.934780 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:27.935175 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:27.935258 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:27.937730 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:27.937737 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:27.937741 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:27.937780 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:27.940538 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:27.941318 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:27.941323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:27.941327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:27.941471 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:27.941847 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:27.941853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:27.941857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:27.942237 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:27.942394 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:27.942710 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 04:33:27.942957 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 04:33:27.948699 2494 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34065
I20260619 04:33:27.948730 2628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34065 every 8 connection(s)
I20260619 04:33:27.949815 2494 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:33:27.951511 2629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37573
I20260619 04:33:27.951699 2629 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:27.952276 2629 heartbeater.cc:511] Master 127.0.97.62:37573 requested a full tablet report, sending...
I20260619 04:33:27.953493 2144 ts_manager.cc:194] Registered new tserver with Master: 26e97c9f758a4e46a1afee4c07288303 (127.0.97.3:34065)
I20260619 04:33:27.954329 2144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51573
I20260619 04:33:27.957661 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 2494
I20260619 04:33:27.957803 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:33:27.958171 388 external_mini_cluster.cc:1428] Setting key 9ba5d0851756a26bb30ac44e8b1285b7
I20260619 04:33:27.960752 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 04:33:27 dist-test-slave-n3tq krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 04:33:27 dist-test-slave-n3tq krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:27.997088 2351 heartbeater.cc:503] Master 127.0.97.62:37573 was elected leader, sending a full tablet report...
I20260619 04:33:28.002146 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:27.989269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46380 (local address 127.0.97.62:37573)
0619 04:33:27.989460 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:27.989466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:27.989648 (+ 182us) server_negotiation.cc:407] Connection header received
0619 04:33:27.989841 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:27.989845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:27.989920 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:27.990003 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:27.991654 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:27.992333 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:27.993547 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:27.993978 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:27.994084 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:27.997552 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:27.997557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:27.997560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:27.997596 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:28.000091 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:28.000724 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:28.000729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:28.000731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:28.000838 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:28.001096 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:28.001099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:28.001101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:28.001331 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:28.001494 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:28.001721 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 04:33:28.001891 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 04:33:28.005656 2144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:33:28.012022 2144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:33:28.042485 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.023080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43405 (local address 127.0.0.1:58760)
0619 04:33:28.034217 (+ 11137us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:28.034235 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:28.034344 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:28.035426 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:28.035432 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:28.035468 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:28.035924 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.035936 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.037559 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.037565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.039670 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.039683 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.041497 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.041503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.041684 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.041730 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 04:33:28.041865 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 04:33:28.042051 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11000,"mutex_wait_us":3,"thread_start_us":12610,"threads_started":1}
I20260619 04:33:28.042915 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.022893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35787 (local address 127.0.0.1:56914)
0619 04:33:28.023749 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:28.023782 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:28.023940 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:28.036734 (+ 12794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:28.036739 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:28.036761 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:28.037201 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.037213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.039077 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.039081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.040763 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.040774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.042286 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.042292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.042446 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.042482 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 04:33:28.042554 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 04:33:28.042670 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":79,"thread_start_us":454,"threads_started":1}
I20260619 04:33:28.043442 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.023608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58760 (local address 127.0.97.2:43405)
0619 04:33:28.034889 (+ 11281us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:28.034895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:28.034918 (+ 23us) server_negotiation.cc:407] Connection header received
0619 04:33:28.035000 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:28.035006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:28.035207 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:28.035403 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:28.036114 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.037396 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.039818 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.041172 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.041294 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.041493 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:28.042423 (+ 930us) server_negotiation.cc:299] Negotiation successful
0619 04:33:28.043164 (+ 741us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11167,"thread_start_us":11051,"threads_started":1}
I20260619 04:33:28.043689 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.023008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56914 (local address 127.0.97.1:35787)
0619 04:33:28.036258 (+ 13250us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:28.036264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:28.036285 (+ 21us) server_negotiation.cc:407] Connection header received
0619 04:33:28.036361 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:28.036366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:28.036567 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:28.036732 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:28.037453 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.038938 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.040900 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.042085 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.043124 (+ 1039us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.043282 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:28.043334 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 04:33:28.043436 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13130,"thread_start_us":13011,"threads_started":1}
I20260619 04:33:28.044129 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.024116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34065 (local address 127.0.0.1:53488)
0619 04:33:28.035931 (+ 11815us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:28.035948 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:28.036073 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:28.036938 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:28.036942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:28.036967 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:28.037432 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.037442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.039247 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.039252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.041312 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.041325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.043431 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.043437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.043606 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.043637 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 04:33:28.043717 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 04:33:28.043825 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11696,"mutex_wait_us":18,"thread_start_us":10315,"threads_started":1}
I20260619 04:33:28.044903 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.024595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53488 (local address 127.0.97.3:34065)
0619 04:33:28.033952 (+ 9357us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:28.033962 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:28.036182 (+ 2220us) server_negotiation.cc:407] Connection header received
0619 04:33:28.036285 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:28.036292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:28.036555 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:28.036732 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:28.037644 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.039123 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.041459 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.043005 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.043111 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.043313 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:28.044433 (+ 1120us) server_negotiation.cc:299] Negotiation successful
0619 04:33:28.044615 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9169,"thread_start_us":8985,"threads_started":1}
I20260619 04:33:28.046733 2281 tablet_service.cc:1467] Processing CreateTablet for tablet 772ebfb894854060ae6e875bbafb48af (DEFAULT_TABLE table=test-table [id=14a4a303ac5a40b99bc0b8e3302657bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:28.046768 2420 tablet_service.cc:1467] Processing CreateTablet for tablet 772ebfb894854060ae6e875bbafb48af (DEFAULT_TABLE table=test-table [id=14a4a303ac5a40b99bc0b8e3302657bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:28.047847 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 772ebfb894854060ae6e875bbafb48af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:28.047995 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 772ebfb894854060ae6e875bbafb48af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:28.047883 2559 tablet_service.cc:1467] Processing CreateTablet for tablet 772ebfb894854060ae6e875bbafb48af (DEFAULT_TABLE table=test-table [id=14a4a303ac5a40b99bc0b8e3302657bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:28.049124 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 772ebfb894854060ae6e875bbafb48af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:28.059475 2648 tablet_bootstrap.cc:492] T 772ebfb894854060ae6e875bbafb48af P 8e9f2ed18bd4467ea123dc19c92adcaf: Bootstrap starting.
I20260619 04:33:28.061547 2649 tablet_bootstrap.cc:492] T 772ebfb894854060ae6e875bbafb48af P 1dd08036edf54d5293ef71d27be76c4b: Bootstrap starting.
I20260619 04:33:28.063120 2648 tablet_bootstrap.cc:654] T 772ebfb894854060ae6e875bbafb48af P 8e9f2ed18bd4467ea123dc19c92adcaf: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:28.064409 2648 log.cc:824] T 772ebfb894854060ae6e875bbafb48af P 8e9f2ed18bd4467ea123dc19c92adcaf: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:28.064935 2649 tablet_bootstrap.cc:654] T 772ebfb894854060ae6e875bbafb48af P 1dd08036edf54d5293ef71d27be76c4b: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:28.065834 2649 log.cc:824] T 772ebfb894854060ae6e875bbafb48af P 1dd08036edf54d5293ef71d27be76c4b: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:28.067476 2648 tablet_bootstrap.cc:492] T 772ebfb894854060ae6e875bbafb48af P 8e9f2ed18bd4467ea123dc19c92adcaf: No bootstrap required, opened a new log
I20260619 04:33:28.067724 2648 ts_tablet_manager.cc:1397] T 772ebfb894854060ae6e875bbafb48af P 8e9f2ed18bd4467ea123dc19c92adcaf: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 04:33:28.068493 2649 tablet_bootstrap.cc:492] T 772ebfb894854060ae6e875bbafb48af P 1dd08036edf54d5293ef71d27be76c4b: No bootstrap required, opened a new log
I20260619 04:33:28.068729 2649 ts_tablet_manager.cc:1397] T 772ebfb894854060ae6e875bbafb48af P 1dd08036edf54d5293ef71d27be76c4b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.002s
I20260619 04:33:28.071115 2650 tablet_bootstrap.cc:492] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303: Bootstrap starting.
I20260619 04:33:28.073776 2650 tablet_bootstrap.cc:654] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:28.074647 2650 log.cc:824] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:28.076897 2650 tablet_bootstrap.cc:492] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303: No bootstrap required, opened a new log
I20260619 04:33:28.077122 2650 ts_tablet_manager.cc:1397] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 04:33:28.077652 2648 raft_consensus.cc:348] T 772ebfb894854060ae6e875bbafb48af P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.078069 2648 raft_consensus.cc:374] T 772ebfb894854060ae6e875bbafb48af P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:28.078167 2648 raft_consensus.cc:729] T 772ebfb894854060ae6e875bbafb48af P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e9f2ed18bd4467ea123dc19c92adcaf, State: Initialized, Role: FOLLOWER
I20260619 04:33:28.078336 2649 raft_consensus.cc:348] T 772ebfb894854060ae6e875bbafb48af P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.078717 2649 raft_consensus.cc:374] T 772ebfb894854060ae6e875bbafb48af P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:28.078814 2649 raft_consensus.cc:729] T 772ebfb894854060ae6e875bbafb48af P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd08036edf54d5293ef71d27be76c4b, State: Initialized, Role: FOLLOWER
I20260619 04:33:28.078763 2648 consensus_queue.cc:260] T 772ebfb894854060ae6e875bbafb48af P 8e9f2ed18bd4467ea123dc19c92adcaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.079418 2649 consensus_queue.cc:260] T 772ebfb894854060ae6e875bbafb48af P 1dd08036edf54d5293ef71d27be76c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.080390 2490 heartbeater.cc:503] Master 127.0.97.62:37573 was elected leader, sending a full tablet report...
I20260619 04:33:28.081056 2649 ts_tablet_manager.cc:1428] T 772ebfb894854060ae6e875bbafb48af P 1dd08036edf54d5293ef71d27be76c4b: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
I20260619 04:33:28.081492 2648 ts_tablet_manager.cc:1428] T 772ebfb894854060ae6e875bbafb48af P 8e9f2ed18bd4467ea123dc19c92adcaf: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260619 04:33:28.087356 2650 raft_consensus.cc:348] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.087669 2650 raft_consensus.cc:374] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:28.087755 2650 raft_consensus.cc:729] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e97c9f758a4e46a1afee4c07288303, State: Initialized, Role: FOLLOWER
I20260619 04:33:28.088213 2650 consensus_queue.cc:260] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.089066 2629 heartbeater.cc:503] Master 127.0.97.62:37573 was elected leader, sending a full tablet report...
I20260619 04:33:28.089435 2650 ts_tablet_manager.cc:1428] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303: Time spent starting tablet: real 0.012s user 0.002s sys 0.008s
I20260619 04:33:28.149992 2656 raft_consensus.cc:482] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:33:28.150266 2656 raft_consensus.cc:504] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.151814 2656 leader_election.cc:288] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e9f2ed18bd4467ea123dc19c92adcaf (127.0.97.1:35787), 1dd08036edf54d5293ef71d27be76c4b (127.0.97.2:43405)
I20260619 04:33:28.157410 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.152117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35787 (local address 127.0.97.3:52643)
0619 04:33:28.152431 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:28.152472 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:28.152605 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:28.152926 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:28.152931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:28.152958 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:28.153160 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.153170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.154364 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.154368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.155677 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.155688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.156744 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.156749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.156912 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.156934 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 04:33:28.157018 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 04:33:28.157138 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260619 04:33:28.157399 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.152232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52643 (local address 127.0.97.1:35787)
0619 04:33:28.152488 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:28.152493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:28.152524 (+ 31us) server_negotiation.cc:407] Connection header received
0619 04:33:28.152749 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:28.152753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:28.152818 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:28.152890 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:28.153304 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.154236 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.155823 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.156608 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.156722 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.156817 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:28.157057 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 04:33:28.157175 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260619 04:33:28.158051 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.152271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60725 (local address 127.0.97.2:43405)
0619 04:33:28.152481 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:28.152487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:28.153103 (+ 616us) server_negotiation.cc:407] Connection header received
0619 04:33:28.153279 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:28.153284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:28.153355 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:28.153437 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:28.153951 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.154784 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.156379 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.157175 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.157292 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.157386 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:28.157723 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 04:33:28.157846 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 04:33:28.158092 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.152202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43405 (local address 127.0.97.3:60725)
0619 04:33:28.153006 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:28.153020 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:28.153151 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:28.153455 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:28.153459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:28.153480 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:28.153807 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.153818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.154905 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.154910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.156237 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.156248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.157375 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.157381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.157543 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.157576 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 04:33:28.157660 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 04:33:28.157771 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":1,"thread_start_us":487,"threads_started":1}
I20260619 04:33:28.158430 2305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "772ebfb894854060ae6e875bbafb48af" candidate_uuid: "26e97c9f758a4e46a1afee4c07288303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" is_pre_election: true
I20260619 04:33:28.159018 2305 raft_consensus.cc:2454] T 772ebfb894854060ae6e875bbafb48af P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26e97c9f758a4e46a1afee4c07288303 in term 0.
I20260619 04:33:28.159099 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "772ebfb894854060ae6e875bbafb48af" candidate_uuid: "26e97c9f758a4e46a1afee4c07288303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd08036edf54d5293ef71d27be76c4b" is_pre_election: true
I20260619 04:33:28.159551 2440 raft_consensus.cc:2454] T 772ebfb894854060ae6e875bbafb48af P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26e97c9f758a4e46a1afee4c07288303 in term 0.
I20260619 04:33:28.159648 2512 leader_election.cc:302] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26e97c9f758a4e46a1afee4c07288303, 8e9f2ed18bd4467ea123dc19c92adcaf; no voters:
I20260619 04:33:28.160264 2656 raft_consensus.cc:2781] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:33:28.160384 2656 raft_consensus.cc:482] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:33:28.160494 2656 raft_consensus.cc:3037] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:28.162575 2656 raft_consensus.cc:504] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.163100 2656 leader_election.cc:288] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303 [CANDIDATE]: Term 1 election: Requested vote from peers 8e9f2ed18bd4467ea123dc19c92adcaf (127.0.97.1:35787), 1dd08036edf54d5293ef71d27be76c4b (127.0.97.2:43405)
I20260619 04:33:28.163558 2305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "772ebfb894854060ae6e875bbafb48af" candidate_uuid: "26e97c9f758a4e46a1afee4c07288303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf"
I20260619 04:33:28.163555 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "772ebfb894854060ae6e875bbafb48af" candidate_uuid: "26e97c9f758a4e46a1afee4c07288303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd08036edf54d5293ef71d27be76c4b"
I20260619 04:33:28.163757 2305 raft_consensus.cc:3037] T 772ebfb894854060ae6e875bbafb48af P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:28.163757 2440 raft_consensus.cc:3037] T 772ebfb894854060ae6e875bbafb48af P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:28.166798 2440 raft_consensus.cc:2454] T 772ebfb894854060ae6e875bbafb48af P 1dd08036edf54d5293ef71d27be76c4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26e97c9f758a4e46a1afee4c07288303 in term 1.
I20260619 04:33:28.166798 2305 raft_consensus.cc:2454] T 772ebfb894854060ae6e875bbafb48af P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26e97c9f758a4e46a1afee4c07288303 in term 1.
I20260619 04:33:28.167193 2512 leader_election.cc:302] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26e97c9f758a4e46a1afee4c07288303, 8e9f2ed18bd4467ea123dc19c92adcaf; no voters:
I20260619 04:33:28.167568 2656 raft_consensus.cc:2781] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:28.167984 2656 raft_consensus.cc:686] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303 [term 1 LEADER]: Becoming Leader. State: Replica: 26e97c9f758a4e46a1afee4c07288303, State: Running, Role: LEADER
I20260619 04:33:28.168488 2656 consensus_queue.cc:237] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.172636 2143 catalog_manager.cc:5495] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26e97c9f758a4e46a1afee4c07288303 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "26e97c9f758a4e46a1afee4c07288303" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } health_report { overall_health: HEALTHY } } }
W20260619 04:33:28.201252 2630 tablet.cc:2369] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 04:33:28 dist-test-slave-n3tq krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:28.231237 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.218728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37573 (local address 127.0.0.1:46384)
0619 04:33:28.219276 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:28.219295 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:28.219421 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:28.219740 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:28.219742 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:28.220390 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:28.220671 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.220680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.221668 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.221672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.222615 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.222626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.223317 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.223322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.223459 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.224045 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:28.224063 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:28.226245 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:28.228913 (+ 2668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:28.228923 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:28.228929 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:28.229395 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:28.229724 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:28.229727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:28.229729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:28.229856 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:28.230310 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:28.230318 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:28.230531 (+ 213us) client_negotiation.cc:769] Sending connection context
0619 04:33:28.230688 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 04:33:28.230898 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":300,"threads_started":1}
I20260619 04:33:28.231253 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.218712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46384 (local address 127.0.97.62:37573)
0619 04:33:28.218871 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:28.218876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:28.219396 (+ 520us) server_negotiation.cc:407] Connection header received
0619 04:33:28.219557 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:28.219561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:28.219622 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:28.219705 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:28.220818 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.221550 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.222739 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.223144 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.223237 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.226426 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:28.226431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:28.226434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:28.226462 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:28.228782 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:28.229521 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:28.229525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:28.229527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:28.229641 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:28.229936 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:28.229939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:28.229941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:28.230185 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:28.230289 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:28.230772 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 04:33:28.231012 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":57}
W20260619 04:33:28.234262 2491 tablet.cc:2369] T 772ebfb894854060ae6e875bbafb48af P 1dd08036edf54d5293ef71d27be76c4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:33:28.235834 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46384:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 04:33:28.236816 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260619 04:33:28.240747 2352 tablet.cc:2369] T 772ebfb894854060ae6e875bbafb48af P 8e9f2ed18bd4467ea123dc19c92adcaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:33:28.242942 2281 tablet_service.cc:1467] Processing CreateTablet for tablet a8673ba6d5364601ac91e07553ffa445 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9c93a4d8f9cc4d95965a20ce23c86b79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 04:33:28.243218 2559 tablet_service.cc:1467] Processing CreateTablet for tablet a8673ba6d5364601ac91e07553ffa445 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9c93a4d8f9cc4d95965a20ce23c86b79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 04:33:28.243086 2420 tablet_service.cc:1467] Processing CreateTablet for tablet a8673ba6d5364601ac91e07553ffa445 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9c93a4d8f9cc4d95965a20ce23c86b79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 04:33:28.243820 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8673ba6d5364601ac91e07553ffa445. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:28.243923 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8673ba6d5364601ac91e07553ffa445. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:28.243927 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8673ba6d5364601ac91e07553ffa445. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:28.249512 2648 tablet_bootstrap.cc:492] T a8673ba6d5364601ac91e07553ffa445 P 8e9f2ed18bd4467ea123dc19c92adcaf: Bootstrap starting.
I20260619 04:33:28.250801 2648 tablet_bootstrap.cc:654] T a8673ba6d5364601ac91e07553ffa445 P 8e9f2ed18bd4467ea123dc19c92adcaf: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:28.250906 2650 tablet_bootstrap.cc:492] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303: Bootstrap starting.
I20260619 04:33:28.251538 2649 tablet_bootstrap.cc:492] T a8673ba6d5364601ac91e07553ffa445 P 1dd08036edf54d5293ef71d27be76c4b: Bootstrap starting.
I20260619 04:33:28.252517 2650 tablet_bootstrap.cc:654] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:28.253006 2649 tablet_bootstrap.cc:654] T a8673ba6d5364601ac91e07553ffa445 P 1dd08036edf54d5293ef71d27be76c4b: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:28.254803 2650 tablet_bootstrap.cc:492] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303: No bootstrap required, opened a new log
I20260619 04:33:28.254961 2650 ts_tablet_manager.cc:1397] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 04:33:28.254997 2649 tablet_bootstrap.cc:492] T a8673ba6d5364601ac91e07553ffa445 P 1dd08036edf54d5293ef71d27be76c4b: No bootstrap required, opened a new log
I20260619 04:33:28.255117 2649 ts_tablet_manager.cc:1397] T a8673ba6d5364601ac91e07553ffa445 P 1dd08036edf54d5293ef71d27be76c4b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260619 04:33:28.255208 2648 tablet_bootstrap.cc:492] T a8673ba6d5364601ac91e07553ffa445 P 8e9f2ed18bd4467ea123dc19c92adcaf: No bootstrap required, opened a new log
I20260619 04:33:28.255327 2648 ts_tablet_manager.cc:1397] T a8673ba6d5364601ac91e07553ffa445 P 8e9f2ed18bd4467ea123dc19c92adcaf: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260619 04:33:28.255759 2649 raft_consensus.cc:348] T a8673ba6d5364601ac91e07553ffa445 P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.255751 2650 raft_consensus.cc:348] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.255939 2649 raft_consensus.cc:374] T a8673ba6d5364601ac91e07553ffa445 P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:28.255983 2650 raft_consensus.cc:374] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:28.256023 2649 raft_consensus.cc:729] T a8673ba6d5364601ac91e07553ffa445 P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd08036edf54d5293ef71d27be76c4b, State: Initialized, Role: FOLLOWER
I20260619 04:33:28.256068 2650 raft_consensus.cc:729] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e97c9f758a4e46a1afee4c07288303, State: Initialized, Role: FOLLOWER
I20260619 04:33:28.256016 2648 raft_consensus.cc:348] T a8673ba6d5364601ac91e07553ffa445 P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.256223 2648 raft_consensus.cc:374] T a8673ba6d5364601ac91e07553ffa445 P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:28.256314 2648 raft_consensus.cc:729] T a8673ba6d5364601ac91e07553ffa445 P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e9f2ed18bd4467ea123dc19c92adcaf, State: Initialized, Role: FOLLOWER
I20260619 04:33:28.256237 2649 consensus_queue.cc:260] T a8673ba6d5364601ac91e07553ffa445 P 1dd08036edf54d5293ef71d27be76c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.256270 2650 consensus_queue.cc:260] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.256799 2649 ts_tablet_manager.cc:1428] T a8673ba6d5364601ac91e07553ffa445 P 1dd08036edf54d5293ef71d27be76c4b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 04:33:28.256842 2650 ts_tablet_manager.cc:1428] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 04:33:28.256692 2656 tablet_replica.cc:439] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } } }
I20260619 04:33:28.256734 2655 tablet_replica.cc:439] T a8673ba6d5364601ac91e07553ffa445 P 1dd08036edf54d5293ef71d27be76c4b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } } }
I20260619 04:33:28.257001 2656 tablet_replica.cc:442] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:33:28.257058 2655 tablet_replica.cc:442] T a8673ba6d5364601ac91e07553ffa445 P 1dd08036edf54d5293ef71d27be76c4b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:33:28.257265 2648 consensus_queue.cc:260] T a8673ba6d5364601ac91e07553ffa445 P 8e9f2ed18bd4467ea123dc19c92adcaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.257790 2648 ts_tablet_manager.cc:1428] T a8673ba6d5364601ac91e07553ffa445 P 8e9f2ed18bd4467ea123dc19c92adcaf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 04:33:28.257673 2654 tablet_replica.cc:439] T a8673ba6d5364601ac91e07553ffa445 P 8e9f2ed18bd4467ea123dc19c92adcaf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } } }
I20260619 04:33:28.258021 2654 tablet_replica.cc:442] T a8673ba6d5364601ac91e07553ffa445 P 8e9f2ed18bd4467ea123dc19c92adcaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:33:28.291710 2656 raft_consensus.cc:482] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:33:28.291934 2656 raft_consensus.cc:504] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.292582 2656 leader_election.cc:288] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e9f2ed18bd4467ea123dc19c92adcaf (127.0.97.1:35787), 1dd08036edf54d5293ef71d27be76c4b (127.0.97.2:43405)
I20260619 04:33:28.293174 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8673ba6d5364601ac91e07553ffa445" candidate_uuid: "26e97c9f758a4e46a1afee4c07288303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd08036edf54d5293ef71d27be76c4b" is_pre_election: true
I20260619 04:33:28.293174 2305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8673ba6d5364601ac91e07553ffa445" candidate_uuid: "26e97c9f758a4e46a1afee4c07288303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" is_pre_election: true
I20260619 04:33:28.293444 2305 raft_consensus.cc:2454] T a8673ba6d5364601ac91e07553ffa445 P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26e97c9f758a4e46a1afee4c07288303 in term 0.
I20260619 04:33:28.293478 2440 raft_consensus.cc:2454] T a8673ba6d5364601ac91e07553ffa445 P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26e97c9f758a4e46a1afee4c07288303 in term 0.
I20260619 04:33:28.293864 2512 leader_election.cc:302] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26e97c9f758a4e46a1afee4c07288303, 8e9f2ed18bd4467ea123dc19c92adcaf; no voters:
I20260619 04:33:28.294176 2656 raft_consensus.cc:2781] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:33:28.294282 2656 raft_consensus.cc:482] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:33:28.294377 2656 raft_consensus.cc:3037] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:28.296276 2656 raft_consensus.cc:504] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.296793 2656 leader_election.cc:288] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303 [CANDIDATE]: Term 1 election: Requested vote from peers 8e9f2ed18bd4467ea123dc19c92adcaf (127.0.97.1:35787), 1dd08036edf54d5293ef71d27be76c4b (127.0.97.2:43405)
I20260619 04:33:28.297194 2305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8673ba6d5364601ac91e07553ffa445" candidate_uuid: "26e97c9f758a4e46a1afee4c07288303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf"
I20260619 04:33:28.297214 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8673ba6d5364601ac91e07553ffa445" candidate_uuid: "26e97c9f758a4e46a1afee4c07288303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd08036edf54d5293ef71d27be76c4b"
I20260619 04:33:28.297389 2305 raft_consensus.cc:3037] T a8673ba6d5364601ac91e07553ffa445 P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:28.297405 2440 raft_consensus.cc:3037] T a8673ba6d5364601ac91e07553ffa445 P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:28.299242 2305 raft_consensus.cc:2454] T a8673ba6d5364601ac91e07553ffa445 P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26e97c9f758a4e46a1afee4c07288303 in term 1.
I20260619 04:33:28.299320 2440 raft_consensus.cc:2454] T a8673ba6d5364601ac91e07553ffa445 P 1dd08036edf54d5293ef71d27be76c4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26e97c9f758a4e46a1afee4c07288303 in term 1.
I20260619 04:33:28.299641 2512 leader_election.cc:302] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26e97c9f758a4e46a1afee4c07288303, 8e9f2ed18bd4467ea123dc19c92adcaf; no voters:
I20260619 04:33:28.299863 2656 raft_consensus.cc:2781] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:28.300058 2656 raft_consensus.cc:686] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303 [term 1 LEADER]: Becoming Leader. State: Replica: 26e97c9f758a4e46a1afee4c07288303, State: Running, Role: LEADER
I20260619 04:33:28.300279 2656 consensus_queue.cc:237] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.301806 2658 tablet_replica.cc:439] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303: TxnStatusTablet state changed. Reason: New leader 26e97c9f758a4e46a1afee4c07288303. Latest consensus state: current_term: 1 leader_uuid: "26e97c9f758a4e46a1afee4c07288303" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } } }
I20260619 04:33:28.302181 2658 tablet_replica.cc:442] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303: This TxnStatusTablet replica's current role is: LEADER
I20260619 04:33:28.302865 2670 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 04:33:28.303082 2670 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 04:33:28.302982 2143 catalog_manager.cc:5495] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26e97c9f758a4e46a1afee4c07288303 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "26e97c9f758a4e46a1afee4c07288303" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } health_report { overall_health: HEALTHY } } }
I20260619 04:33:28.326069 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.319476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53502 (local address 127.0.97.3:34065)
0619 04:33:28.319661 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:28.319667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:28.319686 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:33:28.319770 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:28.319775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:28.319868 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:28.320018 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:33:28.320388 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.321504 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.323410 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.324101 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.324231 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.324785 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:33:28.325449 (+ 664us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:33:28.325590 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:28.325698 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 04:33:28.325778 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 04:33:28.326167 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.319159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34065 (local address 127.0.0.1:53502)
0619 04:33:28.319396 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:28.319417 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:28.319566 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:28.320043 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:28.320048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:28.320059 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:33:28.320250 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.320260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.321652 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.321657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.323267 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.323285 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.324252 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.324258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.324510 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.324584 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:33:28.325567 (+ 983us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:33:28.325589 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 04:33:28.325664 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 04:33:28.325822 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 04:33:28.335119 2440 raft_consensus.cc:1261] T a8673ba6d5364601ac91e07553ffa445 P 1dd08036edf54d5293ef71d27be76c4b [term 1 FOLLOWER]: Refusing update from remote peer 26e97c9f758a4e46a1afee4c07288303: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:33:28.335119 2305 raft_consensus.cc:1261] T a8673ba6d5364601ac91e07553ffa445 P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 FOLLOWER]: Refusing update from remote peer 26e97c9f758a4e46a1afee4c07288303: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:33:28.336110 2658 consensus_queue.cc:1035] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:33:28.336551 2656 consensus_queue.cc:1035] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:33:28.353948 2655 tablet_replica.cc:439] T a8673ba6d5364601ac91e07553ffa445 P 1dd08036edf54d5293ef71d27be76c4b: TxnStatusTablet state changed. Reason: New leader 26e97c9f758a4e46a1afee4c07288303. Latest consensus state: current_term: 1 leader_uuid: "26e97c9f758a4e46a1afee4c07288303" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } } }
I20260619 04:33:28.354291 2655 tablet_replica.cc:442] T a8673ba6d5364601ac91e07553ffa445 P 1dd08036edf54d5293ef71d27be76c4b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:33:28.358469 2654 tablet_replica.cc:439] T a8673ba6d5364601ac91e07553ffa445 P 8e9f2ed18bd4467ea123dc19c92adcaf: TxnStatusTablet state changed. Reason: New leader 26e97c9f758a4e46a1afee4c07288303. Latest consensus state: current_term: 1 leader_uuid: "26e97c9f758a4e46a1afee4c07288303" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } } }
I20260619 04:33:28.358762 2654 tablet_replica.cc:442] T a8673ba6d5364601ac91e07553ffa445 P 8e9f2ed18bd4467ea123dc19c92adcaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:33:28.358865 2673 mvcc.cc:205] Tried to move back new op lower bound from 7298431419730399232 to 7298431419599970304. Current Snapshot: MvccSnapshot[applied={T|T < 7298431419730399232}]
I20260619 04:33:28.360672 2675 mvcc.cc:205] Tried to move back new op lower bound from 7298431419730399232 to 7298431419599970304. Current Snapshot: MvccSnapshot[applied={T|T < 7298431419730399232}]
I20260619 04:33:28.362823 2674 mvcc.cc:205] Tried to move back new op lower bound from 7298431419730399232 to 7298431419599970304. Current Snapshot: MvccSnapshot[applied={T|T < 7298431419730399232}]
I20260619 04:33:28.369495 2658 tablet_replica.cc:439] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26e97c9f758a4e46a1afee4c07288303" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } } }
I20260619 04:33:28.369791 2658 tablet_replica.cc:442] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303: This TxnStatusTablet replica's current role is: LEADER
I20260619 04:33:28.372562 2658 tablet_replica.cc:439] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26e97c9f758a4e46a1afee4c07288303" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } } }
I20260619 04:33:28.372800 2658 tablet_replica.cc:442] T a8673ba6d5364601ac91e07553ffa445 P 26e97c9f758a4e46a1afee4c07288303: This TxnStatusTablet replica's current role is: LEADER
I20260619 04:33:28.372908 2654 tablet_replica.cc:439] T a8673ba6d5364601ac91e07553ffa445 P 8e9f2ed18bd4467ea123dc19c92adcaf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26e97c9f758a4e46a1afee4c07288303" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } } }
I20260619 04:33:28.373157 2654 tablet_replica.cc:442] T a8673ba6d5364601ac91e07553ffa445 P 8e9f2ed18bd4467ea123dc19c92adcaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:33:28.379770 2655 tablet_replica.cc:439] T a8673ba6d5364601ac91e07553ffa445 P 1dd08036edf54d5293ef71d27be76c4b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26e97c9f758a4e46a1afee4c07288303" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } } }
I20260619 04:33:28.379980 2655 tablet_replica.cc:442] T a8673ba6d5364601ac91e07553ffa445 P 1dd08036edf54d5293ef71d27be76c4b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:33:28.405371 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.398089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.0.97.3:34065)
0619 04:33:28.398254 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:28.398260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:28.398279 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:33:28.398350 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:28.398355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:28.398444 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:28.398547 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:33:28.399124 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.400306 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.402336 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.403020 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.403187 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.403782 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:33:28.403922 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:33:28.404007 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:28.404940 (+ 933us) server_negotiation.cc:299] Negotiation successful
0619 04:33:28.405043 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:33:28.423791 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.417372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54461 (local address 127.0.97.3:34065)
0619 04:33:28.417557 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:28.417563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:28.417774 (+ 211us) server_negotiation.cc:407] Connection header received
0619 04:33:28.417959 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:28.417968 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:28.418044 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:28.418128 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:33:28.418488 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.419578 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.421591 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.422244 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.422359 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.422881 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:33:28.423047 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:33:28.423168 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:28.423308 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 04:33:28.423384 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":57}
I20260619 04:33:28.423856 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.417321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34065 (local address 127.0.97.3:54461)
0619 04:33:28.417682 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:28.417702 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:28.417828 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:28.418153 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:28.418158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:28.418169 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:33:28.418362 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.418371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.419717 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.419722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.421444 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.421465 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.422401 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.422407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.422584 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.422664 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:33:28.423187 (+ 523us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:33:28.423198 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 04:33:28.423285 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 04:33:28.423458 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":1}
I20260619 04:33:28.435758 2304 raft_consensus.cc:1261] T 772ebfb894854060ae6e875bbafb48af P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 FOLLOWER]: Refusing update from remote peer 26e97c9f758a4e46a1afee4c07288303: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:33:28.435845 2439 raft_consensus.cc:1261] T 772ebfb894854060ae6e875bbafb48af P 1dd08036edf54d5293ef71d27be76c4b [term 1 FOLLOWER]: Refusing update from remote peer 26e97c9f758a4e46a1afee4c07288303: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:33:28.436391 2658 consensus_queue.cc:1035] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:33:28.437211 2656 consensus_queue.cc:1035] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:33:28.544487 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:33:28.544803 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:33:28.548097 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482: Sending DeleteTablet for 3 replicas of tablet 772ebfb894854060ae6e875bbafb48af
I20260619 04:33:28.549142 2281 tablet_service.cc:1514] Processing DeleteTablet for tablet 772ebfb894854060ae6e875bbafb48af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56914
I20260619 04:33:28.549455 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet 772ebfb894854060ae6e875bbafb48af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58760
I20260619 04:33:28.549950 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet 772ebfb894854060ae6e875bbafb48af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53488
I20260619 04:33:28.551738 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:33:28.552435 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:33:28.562009 2419 tablet_service.cc:1467] Processing CreateTablet for tablet ed1c411f6b114a549412ace57ef5e62a (DEFAULT_TABLE table=test-table [id=dd88fcffbbf04870bbb419f1e7e75e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:28.562631 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed1c411f6b114a549412ace57ef5e62a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:28.565420 2559 tablet_service.cc:1467] Processing CreateTablet for tablet ed1c411f6b114a549412ace57ef5e62a (DEFAULT_TABLE table=test-table [id=dd88fcffbbf04870bbb419f1e7e75e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:28.565841 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed1c411f6b114a549412ace57ef5e62a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:28.569202 2280 tablet_service.cc:1467] Processing CreateTablet for tablet ed1c411f6b114a549412ace57ef5e62a (DEFAULT_TABLE table=test-table [id=dd88fcffbbf04870bbb419f1e7e75e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:28.569589 2689 tablet_replica.cc:330] T 772ebfb894854060ae6e875bbafb48af P 1dd08036edf54d5293ef71d27be76c4b: stopping tablet replica
I20260619 04:33:28.569718 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed1c411f6b114a549412ace57ef5e62a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:28.570072 2689 raft_consensus.cc:2229] T 772ebfb894854060ae6e875bbafb48af P 1dd08036edf54d5293ef71d27be76c4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:33:28.570468 2689 raft_consensus.cc:2258] T 772ebfb894854060ae6e875bbafb48af P 1dd08036edf54d5293ef71d27be76c4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:28.572367 2689 ts_tablet_manager.cc:1905] T 772ebfb894854060ae6e875bbafb48af P 1dd08036edf54d5293ef71d27be76c4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:28.572862 2688 tablet_replica.cc:330] T 772ebfb894854060ae6e875bbafb48af P 8e9f2ed18bd4467ea123dc19c92adcaf: stopping tablet replica
I20260619 04:33:28.573254 2688 raft_consensus.cc:2229] T 772ebfb894854060ae6e875bbafb48af P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:33:28.573589 2688 raft_consensus.cc:2258] T 772ebfb894854060ae6e875bbafb48af P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:28.578181 2649 tablet_bootstrap.cc:492] T ed1c411f6b114a549412ace57ef5e62a P 1dd08036edf54d5293ef71d27be76c4b: Bootstrap starting.
I20260619 04:33:28.579800 2649 tablet_bootstrap.cc:654] T ed1c411f6b114a549412ace57ef5e62a P 1dd08036edf54d5293ef71d27be76c4b: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:28.588033 2688 ts_tablet_manager.cc:1905] T 772ebfb894854060ae6e875bbafb48af P 8e9f2ed18bd4467ea123dc19c92adcaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:28.593816 2650 tablet_bootstrap.cc:492] T ed1c411f6b114a549412ace57ef5e62a P 26e97c9f758a4e46a1afee4c07288303: Bootstrap starting.
I20260619 04:33:28.595687 2650 tablet_bootstrap.cc:654] T ed1c411f6b114a549412ace57ef5e62a P 26e97c9f758a4e46a1afee4c07288303: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:28.600015 2649 tablet_bootstrap.cc:492] T ed1c411f6b114a549412ace57ef5e62a P 1dd08036edf54d5293ef71d27be76c4b: No bootstrap required, opened a new log
I20260619 04:33:28.600219 2649 ts_tablet_manager.cc:1397] T ed1c411f6b114a549412ace57ef5e62a P 1dd08036edf54d5293ef71d27be76c4b: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.004s
I20260619 04:33:28.600384 2648 tablet_bootstrap.cc:492] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf: Bootstrap starting.
I20260619 04:33:28.601080 2649 raft_consensus.cc:348] T ed1c411f6b114a549412ace57ef5e62a P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.601320 2649 raft_consensus.cc:374] T ed1c411f6b114a549412ace57ef5e62a P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:28.601380 2649 raft_consensus.cc:729] T ed1c411f6b114a549412ace57ef5e62a P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd08036edf54d5293ef71d27be76c4b, State: Initialized, Role: FOLLOWER
I20260619 04:33:28.601666 2649 consensus_queue.cc:260] T ed1c411f6b114a549412ace57ef5e62a P 1dd08036edf54d5293ef71d27be76c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.602324 2648 tablet_bootstrap.cc:654] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:28.604601 2649 ts_tablet_manager.cc:1428] T ed1c411f6b114a549412ace57ef5e62a P 1dd08036edf54d5293ef71d27be76c4b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 04:33:28.617087 2689 ts_tablet_manager.cc:1918] T 772ebfb894854060ae6e875bbafb48af P 1dd08036edf54d5293ef71d27be76c4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:33:28.617300 2689 log.cc:1196] T 772ebfb894854060ae6e875bbafb48af P 1dd08036edf54d5293ef71d27be76c4b: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/772ebfb894854060ae6e875bbafb48af
I20260619 04:33:28.617784 2689 ts_tablet_manager.cc:1939] T 772ebfb894854060ae6e875bbafb48af P 1dd08036edf54d5293ef71d27be76c4b: Deleting consensus metadata
I20260619 04:33:28.619679 2128 catalog_manager.cc:4841] TS 1dd08036edf54d5293ef71d27be76c4b (127.0.97.2:43405): tablet 772ebfb894854060ae6e875bbafb48af (table test-table [id=14a4a303ac5a40b99bc0b8e3302657bc]) successfully deleted
I20260619 04:33:28.622344 2688 ts_tablet_manager.cc:1918] T 772ebfb894854060ae6e875bbafb48af P 8e9f2ed18bd4467ea123dc19c92adcaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:33:28.622632 2688 log.cc:1196] T 772ebfb894854060ae6e875bbafb48af P 8e9f2ed18bd4467ea123dc19c92adcaf: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-0/wal/wals/772ebfb894854060ae6e875bbafb48af
I20260619 04:33:28.623277 2688 ts_tablet_manager.cc:1939] T 772ebfb894854060ae6e875bbafb48af P 8e9f2ed18bd4467ea123dc19c92adcaf: Deleting consensus metadata
I20260619 04:33:28.623920 2650 tablet_bootstrap.cc:492] T ed1c411f6b114a549412ace57ef5e62a P 26e97c9f758a4e46a1afee4c07288303: No bootstrap required, opened a new log
I20260619 04:33:28.624102 2650 ts_tablet_manager.cc:1397] T ed1c411f6b114a549412ace57ef5e62a P 26e97c9f758a4e46a1afee4c07288303: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.003s
I20260619 04:33:28.625244 2648 tablet_bootstrap.cc:492] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf: No bootstrap required, opened a new log
I20260619 04:33:28.625391 2648 ts_tablet_manager.cc:1397] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.005s
I20260619 04:33:28.625563 2130 catalog_manager.cc:4841] TS 8e9f2ed18bd4467ea123dc19c92adcaf (127.0.97.1:35787): tablet 772ebfb894854060ae6e875bbafb48af (table test-table [id=14a4a303ac5a40b99bc0b8e3302657bc]) successfully deleted
I20260619 04:33:28.626066 2648 raft_consensus.cc:348] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.626286 2648 raft_consensus.cc:374] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:28.626346 2648 raft_consensus.cc:729] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e9f2ed18bd4467ea123dc19c92adcaf, State: Initialized, Role: FOLLOWER
I20260619 04:33:28.626567 2648 consensus_queue.cc:260] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.627090 2648 ts_tablet_manager.cc:1428] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 04:33:28.627951 2650 raft_consensus.cc:348] T ed1c411f6b114a549412ace57ef5e62a P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.628197 2650 raft_consensus.cc:374] T ed1c411f6b114a549412ace57ef5e62a P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:28.628271 2650 raft_consensus.cc:729] T ed1c411f6b114a549412ace57ef5e62a P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e97c9f758a4e46a1afee4c07288303, State: Initialized, Role: FOLLOWER
I20260619 04:33:28.628518 2650 consensus_queue.cc:260] T ed1c411f6b114a549412ace57ef5e62a P 26e97c9f758a4e46a1afee4c07288303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.629038 2650 ts_tablet_manager.cc:1428] T ed1c411f6b114a549412ace57ef5e62a P 26e97c9f758a4e46a1afee4c07288303: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 04:33:28.630038 2690 tablet_replica.cc:330] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303: stopping tablet replica
I20260619 04:33:28.630525 2690 raft_consensus.cc:2229] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:33:28.631129 2690 raft_consensus.cc:2258] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:28.633263 2690 ts_tablet_manager.cc:1905] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:28.634918 2654 raft_consensus.cc:482] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:33:28.635083 2654 raft_consensus.cc:504] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.636497 2654 leader_election.cc:288] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dd08036edf54d5293ef71d27be76c4b (127.0.97.2:43405), 26e97c9f758a4e46a1afee4c07288303 (127.0.97.3:34065)
I20260619 04:33:28.640738 2690 ts_tablet_manager.cc:1918] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:33:28.640900 2690 log.cc:1196] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/772ebfb894854060ae6e875bbafb48af
I20260619 04:33:28.641397 2690 ts_tablet_manager.cc:1939] T 772ebfb894854060ae6e875bbafb48af P 26e97c9f758a4e46a1afee4c07288303: Deleting consensus metadata
I20260619 04:33:28.643004 2130 catalog_manager.cc:4841] TS 26e97c9f758a4e46a1afee4c07288303 (127.0.97.3:34065): tablet 772ebfb894854060ae6e875bbafb48af (table test-table [id=14a4a303ac5a40b99bc0b8e3302657bc]) successfully deleted
I20260619 04:33:28.650133 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.636852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56813 (local address 127.0.97.2:43405)
0619 04:33:28.637027 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:28.637034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:28.643804 (+ 6770us) server_negotiation.cc:407] Connection header received
0619 04:33:28.644113 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:28.644119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:28.644211 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:28.644298 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:28.645020 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.646111 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.648397 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.649211 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.649328 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.649427 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:28.649722 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 04:33:28.649842 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 04:33:28.650282 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.636715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43405 (local address 127.0.97.1:56813)
0619 04:33:28.643683 (+ 6968us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:28.643705 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:28.643931 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:28.644325 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:28.644329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:28.644351 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:28.644813 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.644829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.646250 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.646255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.648241 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.648250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.649407 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.649413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.649577 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.649601 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 04:33:28.649681 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 04:33:28.649806 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6791,"mutex_wait_us":22,"thread_start_us":7001,"threads_started":1}
I20260619 04:33:28.650727 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.637551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51555 (local address 127.0.97.3:34065)
0619 04:33:28.637726 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:28.637733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:28.644081 (+ 6348us) server_negotiation.cc:407] Connection header received
0619 04:33:28.644242 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:28.644247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:28.644317 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:28.644398 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:28.645093 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.645843 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.648363 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.649640 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.649757 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.649842 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:28.650336 (+ 494us) server_negotiation.cc:299] Negotiation successful
0619 04:33:28.650460 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 04:33:28.650961 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed1c411f6b114a549412ace57ef5e62a" candidate_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd08036edf54d5293ef71d27be76c4b" is_pre_election: true
I20260619 04:33:28.651333 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.637104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34065 (local address 127.0.97.1:51555)
0619 04:33:28.643987 (+ 6883us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:28.644004 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:28.644119 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:28.644534 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:28.644538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:28.644562 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:28.644963 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.644972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.645983 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.645988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.648195 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.648213 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.649995 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.650001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.650176 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.650209 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 04:33:28.650877 (+ 668us) client_negotiation.cc:240] Negotiation successful
0619 04:33:28.651005 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6779,"mutex_wait_us":28,"thread_start_us":6224,"threads_started":1}
I20260619 04:33:28.651266 2439 raft_consensus.cc:2454] T ed1c411f6b114a549412ace57ef5e62a P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e9f2ed18bd4467ea123dc19c92adcaf in term 0.
I20260619 04:33:28.652539 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed1c411f6b114a549412ace57ef5e62a" candidate_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26e97c9f758a4e46a1afee4c07288303" is_pre_election: true
I20260619 04:33:28.652629 2231 leader_election.cc:302] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd08036edf54d5293ef71d27be76c4b, 8e9f2ed18bd4467ea123dc19c92adcaf; no voters:
I20260619 04:33:28.652940 2579 raft_consensus.cc:2454] T ed1c411f6b114a549412ace57ef5e62a P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e9f2ed18bd4467ea123dc19c92adcaf in term 0.
I20260619 04:33:28.653017 2654 raft_consensus.cc:2781] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:33:28.653123 2654 raft_consensus.cc:482] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:33:28.653213 2654 raft_consensus.cc:3037] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:28.655421 2654 raft_consensus.cc:504] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.655964 2654 leader_election.cc:288] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf [CANDIDATE]: Term 1 election: Requested vote from peers 1dd08036edf54d5293ef71d27be76c4b (127.0.97.2:43405), 26e97c9f758a4e46a1afee4c07288303 (127.0.97.3:34065)
I20260619 04:33:28.656402 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed1c411f6b114a549412ace57ef5e62a" candidate_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd08036edf54d5293ef71d27be76c4b"
I20260619 04:33:28.656425 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed1c411f6b114a549412ace57ef5e62a" candidate_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26e97c9f758a4e46a1afee4c07288303"
I20260619 04:33:28.656627 2579 raft_consensus.cc:3037] T ed1c411f6b114a549412ace57ef5e62a P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:28.656641 2439 raft_consensus.cc:3037] T ed1c411f6b114a549412ace57ef5e62a P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:28.659482 2439 raft_consensus.cc:2454] T ed1c411f6b114a549412ace57ef5e62a P 1dd08036edf54d5293ef71d27be76c4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e9f2ed18bd4467ea123dc19c92adcaf in term 1.
I20260619 04:33:28.659719 2579 raft_consensus.cc:2454] T ed1c411f6b114a549412ace57ef5e62a P 26e97c9f758a4e46a1afee4c07288303 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e9f2ed18bd4467ea123dc19c92adcaf in term 1.
I20260619 04:33:28.659888 2231 leader_election.cc:302] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd08036edf54d5293ef71d27be76c4b, 8e9f2ed18bd4467ea123dc19c92adcaf; no voters:
I20260619 04:33:28.660365 2654 raft_consensus.cc:2781] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:28.660782 2654 raft_consensus.cc:686] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 LEADER]: Becoming Leader. State: Replica: 8e9f2ed18bd4467ea123dc19c92adcaf, State: Running, Role: LEADER
I20260619 04:33:28.661168 2654 consensus_queue.cc:237] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:28.664376 2143 catalog_manager.cc:5495] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e9f2ed18bd4467ea123dc19c92adcaf (127.0.97.1). New cstate: current_term: 1 leader_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } health_report { overall_health: UNKNOWN } } }
I20260619 04:33:28.686470 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.680649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.0.97.1:35787)
0619 04:33:28.681173 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:28.681177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:28.681194 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:33:28.681266 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:28.681269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:28.681341 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:28.681480 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:33:28.681849 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.682594 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.683967 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.684408 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.684554 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.685050 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:33:28.685713 (+ 663us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:33:28.685955 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:28.686002 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 04:33:28.686078 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":278,"threads_started":1}
I20260619 04:33:28.701953 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.696186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34065 (local address 127.0.97.1:57007)
0619 04:33:28.696806 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:28.696821 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:28.696935 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:28.697221 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:28.697224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:28.697232 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:33:28.697457 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.697464 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.698835 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.698839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.699992 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.700003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.700973 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.700978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.701091 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.701138 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:33:28.701522 (+ 384us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:33:28.701529 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 04:33:28.701582 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 04:33:28.701680 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":342,"threads_started":1}
I20260619 04:33:28.701985 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.696252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57007 (local address 127.0.97.3:34065)
0619 04:33:28.696430 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:28.696436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:28.696891 (+ 455us) server_negotiation.cc:407] Connection header received
0619 04:33:28.697042 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:28.697047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:28.697123 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:28.697226 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:33:28.697597 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.698694 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.700120 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.700796 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.700910 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.701300 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:33:28.701435 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:33:28.701526 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:28.701621 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 04:33:28.701705 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 04:33:28.710600 2579 raft_consensus.cc:1261] T ed1c411f6b114a549412ace57ef5e62a P 26e97c9f758a4e46a1afee4c07288303 [term 1 FOLLOWER]: Refusing update from remote peer 8e9f2ed18bd4467ea123dc19c92adcaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:33:28.711304 2439 raft_consensus.cc:1261] T ed1c411f6b114a549412ace57ef5e62a P 1dd08036edf54d5293ef71d27be76c4b [term 1 FOLLOWER]: Refusing update from remote peer 8e9f2ed18bd4467ea123dc19c92adcaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:33:28.711342 2696 consensus_queue.cc:1035] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:33:28.712177 2696 consensus_queue.cc:1035] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:33:28.747643 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.741463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40241 (local address 127.0.97.1:35787)
0619 04:33:28.741601 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:28.741606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:28.741675 (+ 69us) server_negotiation.cc:407] Connection header received
0619 04:33:28.741857 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:28.741861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:28.741944 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:28.742016 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:33:28.742814 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.743863 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.745774 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.746174 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.746297 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.746954 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:33:28.747079 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:33:28.747160 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:28.747304 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 04:33:28.747378 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 04:33:28.747776 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:28.741351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35787 (local address 127.0.97.3:40241)
0619 04:33:28.741607 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:28.741628 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:28.741742 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:28.742171 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:28.742177 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:28.742193 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:33:28.742381 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.742389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.744006 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.744011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.745640 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:28.745655 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:28.746531 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:28.746537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:28.746750 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:28.746798 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:33:28.747178 (+ 380us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:33:28.747189 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 04:33:28.747282 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 04:33:28.747434 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
W20260619 04:33:28.748741 2674 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 04:33:30.334831 2142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:33:30.335031 2142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:33:30.338963 2142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482: Sending DeleteTablet for 3 replicas of tablet ed1c411f6b114a549412ace57ef5e62a
I20260619 04:33:30.339627 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet ed1c411f6b114a549412ace57ef5e62a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56914
I20260619 04:33:30.339689 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet ed1c411f6b114a549412ace57ef5e62a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53488
I20260619 04:33:30.339704 2419 tablet_service.cc:1514] Processing DeleteTablet for tablet ed1c411f6b114a549412ace57ef5e62a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58760
I20260619 04:33:30.349337 2730 tablet_replica.cc:330] T ed1c411f6b114a549412ace57ef5e62a P 1dd08036edf54d5293ef71d27be76c4b: stopping tablet replica
I20260619 04:33:30.349660 2730 raft_consensus.cc:2229] T ed1c411f6b114a549412ace57ef5e62a P 1dd08036edf54d5293ef71d27be76c4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:33:30.349967 2730 raft_consensus.cc:2258] T ed1c411f6b114a549412ace57ef5e62a P 1dd08036edf54d5293ef71d27be76c4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:30.351002 2730 ts_tablet_manager.cc:1905] T ed1c411f6b114a549412ace57ef5e62a P 1dd08036edf54d5293ef71d27be76c4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:30.353935 2729 tablet_replica.cc:330] T ed1c411f6b114a549412ace57ef5e62a P 26e97c9f758a4e46a1afee4c07288303: stopping tablet replica
I20260619 04:33:30.354105 2728 tablet_replica.cc:330] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf: stopping tablet replica
I20260619 04:33:30.354125 2729 raft_consensus.cc:2229] T ed1c411f6b114a549412ace57ef5e62a P 26e97c9f758a4e46a1afee4c07288303 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:33:30.354293 2728 raft_consensus.cc:2229] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:33:30.354817 2728 raft_consensus.cc:2258] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:30.354832 2729 raft_consensus.cc:2258] T ed1c411f6b114a549412ace57ef5e62a P 26e97c9f758a4e46a1afee4c07288303 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:30.355683 2728 ts_tablet_manager.cc:1905] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:30.355786 2729 ts_tablet_manager.cc:1905] T ed1c411f6b114a549412ace57ef5e62a P 26e97c9f758a4e46a1afee4c07288303: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:30.358536 2730 ts_tablet_manager.cc:1918] T ed1c411f6b114a549412ace57ef5e62a P 1dd08036edf54d5293ef71d27be76c4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 04:33:30.358736 2730 log.cc:1196] T ed1c411f6b114a549412ace57ef5e62a P 1dd08036edf54d5293ef71d27be76c4b: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/ed1c411f6b114a549412ace57ef5e62a
I20260619 04:33:30.359248 2730 ts_tablet_manager.cc:1939] T ed1c411f6b114a549412ace57ef5e62a P 1dd08036edf54d5293ef71d27be76c4b: Deleting consensus metadata
I20260619 04:33:30.360374 2128 catalog_manager.cc:4841] TS 1dd08036edf54d5293ef71d27be76c4b (127.0.97.2:43405): tablet ed1c411f6b114a549412ace57ef5e62a (table test-table [id=dd88fcffbbf04870bbb419f1e7e75e87]) successfully deleted
I20260619 04:33:30.361548 2728 ts_tablet_manager.cc:1918] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 04:33:30.361694 2728 log.cc:1196] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-0/wal/wals/ed1c411f6b114a549412ace57ef5e62a
I20260619 04:33:30.362075 2728 ts_tablet_manager.cc:1939] T ed1c411f6b114a549412ace57ef5e62a P 8e9f2ed18bd4467ea123dc19c92adcaf: Deleting consensus metadata
I20260619 04:33:30.362285 2729 ts_tablet_manager.cc:1918] T ed1c411f6b114a549412ace57ef5e62a P 26e97c9f758a4e46a1afee4c07288303: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 04:33:30.362390 2729 log.cc:1196] T ed1c411f6b114a549412ace57ef5e62a P 26e97c9f758a4e46a1afee4c07288303: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/ed1c411f6b114a549412ace57ef5e62a
I20260619 04:33:30.362782 2729 ts_tablet_manager.cc:1939] T ed1c411f6b114a549412ace57ef5e62a P 26e97c9f758a4e46a1afee4c07288303: Deleting consensus metadata
I20260619 04:33:30.363101 2130 catalog_manager.cc:4841] TS 8e9f2ed18bd4467ea123dc19c92adcaf (127.0.97.1:35787): tablet ed1c411f6b114a549412ace57ef5e62a (table test-table [id=dd88fcffbbf04870bbb419f1e7e75e87]) successfully deleted
I20260619 04:33:30.363675 2130 catalog_manager.cc:4841] TS 26e97c9f758a4e46a1afee4c07288303 (127.0.97.3:34065): tablet ed1c411f6b114a549412ace57ef5e62a (table test-table [id=dd88fcffbbf04870bbb419f1e7e75e87]) successfully deleted
Jun 19 04:33:30 dist-test-slave-n3tq krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843610, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 04:33:30 dist-test-slave-n3tq krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843610, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:30.394953 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:30.380150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.0.97.62:37573)
0619 04:33:30.380862 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:30.380868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:30.380893 (+ 25us) server_negotiation.cc:407] Connection header received
0619 04:33:30.380979 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:30.380984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:30.381056 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:30.381148 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:30.382382 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:30.383312 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:30.384577 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:30.385226 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:30.385345 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:30.389420 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:30.389428 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:30.389432 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:30.389474 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:30.392655 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:30.393374 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:30.393378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:30.393380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:30.393499 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:30.393814 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:30.393817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:30.393819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:30.394067 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:30.394234 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:30.394534 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 04:33:30.394708 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":436,"threads_started":1}
I20260619 04:33:30.398305 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:33:30.398887 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:33:30.404685 2280 tablet_service.cc:1467] Processing CreateTablet for tablet 838d0f138dac4665822775fa4461c0ec (DEFAULT_TABLE table=test-table [id=02b3c1e95996450181ab0c4fb1af4bd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:30.404722 2419 tablet_service.cc:1467] Processing CreateTablet for tablet 838d0f138dac4665822775fa4461c0ec (DEFAULT_TABLE table=test-table [id=02b3c1e95996450181ab0c4fb1af4bd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:30.405138 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 838d0f138dac4665822775fa4461c0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:30.405161 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 838d0f138dac4665822775fa4461c0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:30.405161 2558 tablet_service.cc:1467] Processing CreateTablet for tablet 838d0f138dac4665822775fa4461c0ec (DEFAULT_TABLE table=test-table [id=02b3c1e95996450181ab0c4fb1af4bd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:30.405560 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 838d0f138dac4665822775fa4461c0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:30.410283 2739 tablet_bootstrap.cc:492] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf: Bootstrap starting.
I20260619 04:33:30.411808 2739 tablet_bootstrap.cc:654] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:30.412763 2741 tablet_bootstrap.cc:492] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303: Bootstrap starting.
I20260619 04:33:30.413400 2740 tablet_bootstrap.cc:492] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b: Bootstrap starting.
I20260619 04:33:30.414894 2741 tablet_bootstrap.cc:654] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:30.414934 2740 tablet_bootstrap.cc:654] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:30.415539 2739 tablet_bootstrap.cc:492] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf: No bootstrap required, opened a new log
I20260619 04:33:30.415679 2739 ts_tablet_manager.cc:1397] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 04:33:30.416844 2739 raft_consensus.cc:348] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:30.417088 2741 tablet_bootstrap.cc:492] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303: No bootstrap required, opened a new log
I20260619 04:33:30.417084 2739 raft_consensus.cc:374] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:30.417171 2739 raft_consensus.cc:729] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e9f2ed18bd4467ea123dc19c92adcaf, State: Initialized, Role: FOLLOWER
I20260619 04:33:30.417217 2741 ts_tablet_manager.cc:1397] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 04:33:30.417399 2739 consensus_queue.cc:260] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:30.417562 2740 tablet_bootstrap.cc:492] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b: No bootstrap required, opened a new log
I20260619 04:33:30.417927 2739 ts_tablet_manager.cc:1428] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 04:33:30.417937 2741 raft_consensus.cc:348] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:30.418143 2741 raft_consensus.cc:374] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:30.418309 2740 ts_tablet_manager.cc:1397] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260619 04:33:30.418397 2741 raft_consensus.cc:729] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e97c9f758a4e46a1afee4c07288303, State: Initialized, Role: FOLLOWER
I20260619 04:33:30.418648 2741 consensus_queue.cc:260] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:30.419010 2740 raft_consensus.cc:348] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:30.419210 2741 ts_tablet_manager.cc:1428] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 04:33:30.419675 2740 raft_consensus.cc:374] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:30.420192 2740 raft_consensus.cc:729] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd08036edf54d5293ef71d27be76c4b, State: Initialized, Role: FOLLOWER
I20260619 04:33:30.420432 2740 consensus_queue.cc:260] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:30.421305 2740 ts_tablet_manager.cc:1428] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260619 04:33:30.496192 2745 raft_consensus.cc:482] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:33:30.496438 2745 raft_consensus.cc:504] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:30.497893 2745 leader_election.cc:288] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e9f2ed18bd4467ea123dc19c92adcaf (127.0.97.1:35787), 26e97c9f758a4e46a1afee4c07288303 (127.0.97.3:34065)
I20260619 04:33:30.502818 2722 raft_consensus.cc:482] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:33:30.503031 2722 raft_consensus.cc:504] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:30.503731 2722 leader_election.cc:288] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dd08036edf54d5293ef71d27be76c4b (127.0.97.2:43405), 26e97c9f758a4e46a1afee4c07288303 (127.0.97.3:34065)
I20260619 04:33:30.504351 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838d0f138dac4665822775fa4461c0ec" candidate_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd08036edf54d5293ef71d27be76c4b" is_pre_election: true
I20260619 04:33:30.504438 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838d0f138dac4665822775fa4461c0ec" candidate_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26e97c9f758a4e46a1afee4c07288303" is_pre_election: true
I20260619 04:33:30.504640 2440 raft_consensus.cc:2454] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e9f2ed18bd4467ea123dc19c92adcaf in term 0.
I20260619 04:33:30.504699 2578 raft_consensus.cc:2454] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e9f2ed18bd4467ea123dc19c92adcaf in term 0.
I20260619 04:33:30.505123 2231 leader_election.cc:302] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd08036edf54d5293ef71d27be76c4b, 8e9f2ed18bd4467ea123dc19c92adcaf; no voters:
I20260619 04:33:30.505414 2722 raft_consensus.cc:2781] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:33:30.505534 2722 raft_consensus.cc:482] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:33:30.505631 2722 raft_consensus.cc:3037] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:30.506322 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:30.498154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35787 (local address 127.0.97.2:43835)
0619 04:33:30.498750 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:30.498770 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:30.498907 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:30.499963 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:30.499968 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:30.499997 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:30.500351 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:30.500364 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:30.501837 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:30.501841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:30.503897 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:30.503911 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:30.505497 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:30.505508 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:30.505694 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:30.505731 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 04:33:30.505820 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 04:33:30.505981 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":44,"thread_start_us":272,"threads_started":1}
I20260619 04:33:30.507268 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:30.498252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43835 (local address 127.0.97.1:35787)
0619 04:33:30.499649 (+ 1397us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:30.499655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:30.499678 (+ 23us) server_negotiation.cc:407] Connection header received
0619 04:33:30.499746 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:30.499751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:30.499820 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:30.500521 (+ 701us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:30.500674 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:30.501718 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:30.504066 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:30.505259 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:30.506554 (+ 1295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:30.506678 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:30.506727 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 04:33:30.506852 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1265,"thread_start_us":1144,"threads_started":1}
I20260619 04:33:30.507438 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:30.498903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44943 (local address 127.0.97.3:34065)
0619 04:33:30.500084 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:30.500088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:30.500107 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:33:30.500158 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:30.500161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:30.500214 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:30.500336 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:30.500862 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:30.501919 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:30.505145 (+ 3226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:30.506474 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:30.506654 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:30.506764 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:30.506993 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 04:33:30.507162 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1073,"thread_start_us":953,"threads_started":1}
I20260619 04:33:30.507875 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:30.498809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34065 (local address 127.0.97.2:44943)
0619 04:33:30.499484 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:30.499502 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:30.499630 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:30.500415 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:30.500419 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:30.500443 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:30.500717 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:30.500725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:30.502048 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:30.502053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:30.504991 (+ 2938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:30.505005 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:30.506702 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:30.506708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:30.506861 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:30.506892 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 04:33:30.506973 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 04:33:30.507095 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":8,"thread_start_us":321,"threads_started":1}
I20260619 04:33:30.507848 2305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838d0f138dac4665822775fa4461c0ec" candidate_uuid: "1dd08036edf54d5293ef71d27be76c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" is_pre_election: true
I20260619 04:33:30.508509 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838d0f138dac4665822775fa4461c0ec" candidate_uuid: "1dd08036edf54d5293ef71d27be76c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26e97c9f758a4e46a1afee4c07288303" is_pre_election: true
I20260619 04:33:30.508553 2722 raft_consensus.cc:504] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:30.508737 2578 raft_consensus.cc:2454] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dd08036edf54d5293ef71d27be76c4b in term 0.
I20260619 04:33:30.508975 2305 raft_consensus.cc:2379] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dd08036edf54d5293ef71d27be76c4b in current term 1: Already voted for candidate 8e9f2ed18bd4467ea123dc19c92adcaf in this term.
I20260619 04:33:30.509163 2373 leader_election.cc:302] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd08036edf54d5293ef71d27be76c4b, 26e97c9f758a4e46a1afee4c07288303; no voters:
I20260619 04:33:30.509327 2722 leader_election.cc:288] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [CANDIDATE]: Term 1 election: Requested vote from peers 1dd08036edf54d5293ef71d27be76c4b (127.0.97.2:43405), 26e97c9f758a4e46a1afee4c07288303 (127.0.97.3:34065)
I20260619 04:33:30.509490 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838d0f138dac4665822775fa4461c0ec" candidate_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd08036edf54d5293ef71d27be76c4b"
I20260619 04:33:30.509579 2745 raft_consensus.cc:2781] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:33:30.509740 2440 raft_consensus.cc:3037] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:30.509732 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838d0f138dac4665822775fa4461c0ec" candidate_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26e97c9f758a4e46a1afee4c07288303"
I20260619 04:33:30.509902 2578 raft_consensus.cc:3037] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:30.512578 2578 raft_consensus.cc:2454] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e9f2ed18bd4467ea123dc19c92adcaf in term 1.
I20260619 04:33:30.512564 2440 raft_consensus.cc:2454] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e9f2ed18bd4467ea123dc19c92adcaf in term 1.
I20260619 04:33:30.512754 2745 raft_consensus.cc:482] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:33:30.512871 2745 raft_consensus.cc:3037] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [term 1 FOLLOWER]: Advancing to term 2
I20260619 04:33:30.512904 2231 leader_election.cc:302] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd08036edf54d5293ef71d27be76c4b, 8e9f2ed18bd4467ea123dc19c92adcaf; no voters:
I20260619 04:33:30.513249 2722 raft_consensus.cc:2781] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:30.513396 2722 raft_consensus.cc:686] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 LEADER]: Becoming Leader. State: Replica: 8e9f2ed18bd4467ea123dc19c92adcaf, State: Running, Role: LEADER
I20260619 04:33:30.513669 2722 consensus_queue.cc:237] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:30.515331 2745 raft_consensus.cc:504] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:30.516103 2745 leader_election.cc:288] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [CANDIDATE]: Term 2 election: Requested vote from peers 8e9f2ed18bd4467ea123dc19c92adcaf (127.0.97.1:35787), 26e97c9f758a4e46a1afee4c07288303 (127.0.97.3:34065)
I20260619 04:33:30.516559 2305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838d0f138dac4665822775fa4461c0ec" candidate_uuid: "1dd08036edf54d5293ef71d27be76c4b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf"
I20260619 04:33:30.516340 2143 catalog_manager.cc:5495] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e9f2ed18bd4467ea123dc19c92adcaf (127.0.97.1). New cstate: current_term: 1 leader_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } health_report { overall_health: UNKNOWN } } }
I20260619 04:33:30.517062 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838d0f138dac4665822775fa4461c0ec" candidate_uuid: "1dd08036edf54d5293ef71d27be76c4b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26e97c9f758a4e46a1afee4c07288303"
I20260619 04:33:30.517256 2578 raft_consensus.cc:3037] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303 [term 1 FOLLOWER]: Advancing to term 2
I20260619 04:33:30.519495 2578 raft_consensus.cc:2454] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dd08036edf54d5293ef71d27be76c4b in term 2.
I20260619 04:33:30.519841 2373 leader_election.cc:302] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dd08036edf54d5293ef71d27be76c4b, 26e97c9f758a4e46a1afee4c07288303; no voters: 8e9f2ed18bd4467ea123dc19c92adcaf
I20260619 04:33:30.520088 2745 raft_consensus.cc:2781] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [term 2 FOLLOWER]: Leader election won for term 2
I20260619 04:33:30.520521 2745 raft_consensus.cc:686] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [term 2 LEADER]: Becoming Leader. State: Replica: 1dd08036edf54d5293ef71d27be76c4b, State: Running, Role: LEADER
I20260619 04:33:30.520881 2745 consensus_queue.cc:237] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:30.523749 2143 catalog_manager.cc:5495] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b reported cstate change: term changed from 1 to 2, leader changed from 8e9f2ed18bd4467ea123dc19c92adcaf (127.0.97.1) to 1dd08036edf54d5293ef71d27be76c4b (127.0.97.2). New cstate: current_term: 2 leader_uuid: "1dd08036edf54d5293ef71d27be76c4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } health_report { overall_health: UNKNOWN } } }
I20260619 04:33:30.543457 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:30.538623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58772 (local address 127.0.97.2:43405)
0619 04:33:30.539073 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:30.539077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:30.539095 (+ 18us) server_negotiation.cc:407] Connection header received
0619 04:33:30.539175 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:30.539178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:30.539245 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:30.539375 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:33:30.539671 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:30.540416 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:30.541656 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:30.542057 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:30.542148 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:30.542451 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:33:30.543009 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:33:30.543120 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:30.543158 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 04:33:30.543214 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":242,"threads_started":1}
I20260619 04:33:30.557293 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:30.552691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56227 (local address 127.0.97.3:34065)
0619 04:33:30.552901 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:30.552907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:30.553193 (+ 286us) server_negotiation.cc:407] Connection header received
0619 04:33:30.553344 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:30.553349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:30.553424 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:30.553502 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:33:30.553906 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:30.554647 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:30.555932 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:30.556317 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:30.556404 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:30.556803 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:33:30.556898 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:33:30.556959 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:30.557039 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 04:33:30.557098 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:33:30.557379 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:30.552539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34065 (local address 127.0.97.2:56227)
0619 04:33:30.553111 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:30.553127 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:30.553232 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:30.553523 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:30.553526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:30.553539 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:33:30.553786 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:30.553793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:30.554751 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:30.554754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:30.555831 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:30.555842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:30.556477 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:30.556484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:30.556624 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:30.556675 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:33:30.556964 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:33:30.556971 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 04:33:30.557019 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 04:33:30.557121 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":319,"threads_started":1}
I20260619 04:33:30.564882 2578 raft_consensus.cc:1261] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303 [term 2 FOLLOWER]: Refusing update from remote peer 1dd08036edf54d5293ef71d27be76c4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260619 04:33:30.564904 2305 raft_consensus.cc:3032] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 LEADER]: Stepping down as leader of term 1
I20260619 04:33:30.565003 2305 raft_consensus.cc:729] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e9f2ed18bd4467ea123dc19c92adcaf, State: Running, Role: LEADER
I20260619 04:33:30.565285 2305 consensus_queue.cc:260] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:30.565618 2305 raft_consensus.cc:3037] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 FOLLOWER]: Advancing to term 2
I20260619 04:33:30.565446 2751 consensus_queue.cc:1035] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:33:30.568830 2305 raft_consensus.cc:1261] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [term 2 FOLLOWER]: Refusing update from remote peer 1dd08036edf54d5293ef71d27be76c4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260619 04:33:30.569425 2745 consensus_queue.cc:1035] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:33:30.569659 2745 consensus_queue.cc:1225] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [LEADER]: Peer 8e9f2ed18bd4467ea123dc19c92adcaf log is divergent from this leader: its last log entry 1.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260619 04:33:30.570487 2304 pending_rounds.cc:76] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf: Aborting all ops after (but not including) 0
I20260619 04:33:30.570600 2304 pending_rounds.cc:97] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260619 04:33:30.570734 2304 raft_consensus.cc:2866] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260619 04:33:30.590922 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:30.585055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54533 (local address 127.0.97.2:43405)
0619 04:33:30.585217 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:30.585225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:30.585719 (+ 494us) server_negotiation.cc:407] Connection header received
0619 04:33:30.585886 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:30.585891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:30.585955 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:30.586024 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:33:30.587001 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:30.587788 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:30.589356 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:30.589809 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:30.589915 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:30.590283 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:33:30.590375 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:33:30.590442 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:30.590573 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 04:33:30.590642 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 04:33:30.590993 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:30.585033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43405 (local address 127.0.97.3:54533)
0619 04:33:30.585629 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:30.585648 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:30.585768 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:30.586084 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:30.586088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:30.586098 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:33:30.586399 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:30.586411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:30.587906 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:30.587912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:30.589241 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:30.589253 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:30.589970 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:30.589977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:30.590109 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:30.590144 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:33:30.590465 (+ 321us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:33:30.590476 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 04:33:30.590545 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 04:33:30.590678 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":310,"threads_started":1}
I20260619 04:33:30.626341 2142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:33:30.626597 2142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:33:30.629595 2142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482: Sending DeleteTablet for 3 replicas of tablet 838d0f138dac4665822775fa4461c0ec
I20260619 04:33:30.630373 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet 838d0f138dac4665822775fa4461c0ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56914
I20260619 04:33:30.630532 2419 tablet_service.cc:1514] Processing DeleteTablet for tablet 838d0f138dac4665822775fa4461c0ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58760
I20260619 04:33:30.630604 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet 838d0f138dac4665822775fa4461c0ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53488
I20260619 04:33:30.630666 2728 tablet_replica.cc:330] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf: stopping tablet replica
I20260619 04:33:30.630844 2728 raft_consensus.cc:2229] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [term 2 FOLLOWER]: Raft consensus shutting down.
I20260619 04:33:30.630887 2730 tablet_replica.cc:330] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b: stopping tablet replica
I20260619 04:33:30.631028 2728 raft_consensus.cc:2258] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf [term 2 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:30.631075 2730 raft_consensus.cc:2229] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [term 2 LEADER]: Raft consensus shutting down.
I20260619 04:33:30.631500 2730 raft_consensus.cc:2258] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:30.631789 2729 tablet_replica.cc:330] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303: stopping tablet replica
I20260619 04:33:30.631961 2729 raft_consensus.cc:2229] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260619 04:33:30.632094 2728 ts_tablet_manager.cc:1905] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:30.632140 2729 raft_consensus.cc:2258] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:30.633152 2729 ts_tablet_manager.cc:1905] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:30.638367 2728 ts_tablet_manager.cc:1918] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.5
I20260619 04:33:30.638520 2728 log.cc:1196] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-0/wal/wals/838d0f138dac4665822775fa4461c0ec
I20260619 04:33:30.639168 2728 ts_tablet_manager.cc:1939] T 838d0f138dac4665822775fa4461c0ec P 8e9f2ed18bd4467ea123dc19c92adcaf: Deleting consensus metadata
I20260619 04:33:30.639524 2730 ts_tablet_manager.cc:1905] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:30.640501 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:33:30.641173 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:33:30.642482 2729 ts_tablet_manager.cc:1918] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.5
I20260619 04:33:30.642642 2729 log.cc:1196] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/838d0f138dac4665822775fa4461c0ec
I20260619 04:33:30.643020 2729 ts_tablet_manager.cc:1939] T 838d0f138dac4665822775fa4461c0ec P 26e97c9f758a4e46a1afee4c07288303: Deleting consensus metadata
I20260619 04:33:30.643622 2130 catalog_manager.cc:4841] TS 8e9f2ed18bd4467ea123dc19c92adcaf (127.0.97.1:35787): tablet 838d0f138dac4665822775fa4461c0ec (table test-table [id=02b3c1e95996450181ab0c4fb1af4bd7]) successfully deleted
I20260619 04:33:30.643973 2130 catalog_manager.cc:4841] TS 26e97c9f758a4e46a1afee4c07288303 (127.0.97.3:34065): tablet 838d0f138dac4665822775fa4461c0ec (table test-table [id=02b3c1e95996450181ab0c4fb1af4bd7]) successfully deleted
I20260619 04:33:30.646749 2730 ts_tablet_manager.cc:1918] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.5
I20260619 04:33:30.646907 2730 log.cc:1196] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/838d0f138dac4665822775fa4461c0ec
I20260619 04:33:30.647329 2730 ts_tablet_manager.cc:1939] T 838d0f138dac4665822775fa4461c0ec P 1dd08036edf54d5293ef71d27be76c4b: Deleting consensus metadata
I20260619 04:33:30.648640 2280 tablet_service.cc:1467] Processing CreateTablet for tablet 259dc80e72194fadbf560486b2911863 (DEFAULT_TABLE table=test-table [id=5a7d7fd9d9d644699ae2550d3a604119]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:30.648764 2419 tablet_service.cc:1467] Processing CreateTablet for tablet 259dc80e72194fadbf560486b2911863 (DEFAULT_TABLE table=test-table [id=5a7d7fd9d9d644699ae2550d3a604119]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:30.649048 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 259dc80e72194fadbf560486b2911863. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:30.649142 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 259dc80e72194fadbf560486b2911863. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:30.653126 2128 catalog_manager.cc:4841] TS 1dd08036edf54d5293ef71d27be76c4b (127.0.97.2:43405): tablet 838d0f138dac4665822775fa4461c0ec (table test-table [id=02b3c1e95996450181ab0c4fb1af4bd7]) successfully deleted
I20260619 04:33:30.653545 2558 tablet_service.cc:1467] Processing CreateTablet for tablet 259dc80e72194fadbf560486b2911863 (DEFAULT_TABLE table=test-table [id=5a7d7fd9d9d644699ae2550d3a604119]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:30.653939 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 259dc80e72194fadbf560486b2911863. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:30.655517 2740 tablet_bootstrap.cc:492] T 259dc80e72194fadbf560486b2911863 P 1dd08036edf54d5293ef71d27be76c4b: Bootstrap starting.
I20260619 04:33:30.657156 2740 tablet_bootstrap.cc:654] T 259dc80e72194fadbf560486b2911863 P 1dd08036edf54d5293ef71d27be76c4b: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:30.658473 2739 tablet_bootstrap.cc:492] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf: Bootstrap starting.
I20260619 04:33:30.660028 2739 tablet_bootstrap.cc:654] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:30.660037 2740 tablet_bootstrap.cc:492] T 259dc80e72194fadbf560486b2911863 P 1dd08036edf54d5293ef71d27be76c4b: No bootstrap required, opened a new log
I20260619 04:33:30.660184 2740 ts_tablet_manager.cc:1397] T 259dc80e72194fadbf560486b2911863 P 1dd08036edf54d5293ef71d27be76c4b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 04:33:30.660925 2740 raft_consensus.cc:348] T 259dc80e72194fadbf560486b2911863 P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:30.661182 2740 raft_consensus.cc:374] T 259dc80e72194fadbf560486b2911863 P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:30.661260 2740 raft_consensus.cc:729] T 259dc80e72194fadbf560486b2911863 P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd08036edf54d5293ef71d27be76c4b, State: Initialized, Role: FOLLOWER
I20260619 04:33:30.661336 2741 tablet_bootstrap.cc:492] T 259dc80e72194fadbf560486b2911863 P 26e97c9f758a4e46a1afee4c07288303: Bootstrap starting.
I20260619 04:33:30.661525 2740 consensus_queue.cc:260] T 259dc80e72194fadbf560486b2911863 P 1dd08036edf54d5293ef71d27be76c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:30.662055 2740 ts_tablet_manager.cc:1428] T 259dc80e72194fadbf560486b2911863 P 1dd08036edf54d5293ef71d27be76c4b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 04:33:30.662660 2741 tablet_bootstrap.cc:654] T 259dc80e72194fadbf560486b2911863 P 26e97c9f758a4e46a1afee4c07288303: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:30.664410 2741 tablet_bootstrap.cc:492] T 259dc80e72194fadbf560486b2911863 P 26e97c9f758a4e46a1afee4c07288303: No bootstrap required, opened a new log
I20260619 04:33:30.664574 2741 ts_tablet_manager.cc:1397] T 259dc80e72194fadbf560486b2911863 P 26e97c9f758a4e46a1afee4c07288303: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 04:33:30.665199 2741 raft_consensus.cc:348] T 259dc80e72194fadbf560486b2911863 P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:30.665405 2741 raft_consensus.cc:374] T 259dc80e72194fadbf560486b2911863 P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:30.665477 2741 raft_consensus.cc:729] T 259dc80e72194fadbf560486b2911863 P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e97c9f758a4e46a1afee4c07288303, State: Initialized, Role: FOLLOWER
I20260619 04:33:30.665700 2741 consensus_queue.cc:260] T 259dc80e72194fadbf560486b2911863 P 26e97c9f758a4e46a1afee4c07288303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:30.666079 2741 ts_tablet_manager.cc:1428] T 259dc80e72194fadbf560486b2911863 P 26e97c9f758a4e46a1afee4c07288303: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 04:33:30.667423 2739 tablet_bootstrap.cc:492] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf: No bootstrap required, opened a new log
I20260619 04:33:30.667593 2739 ts_tablet_manager.cc:1397] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260619 04:33:30.668332 2739 raft_consensus.cc:348] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:30.668648 2739 raft_consensus.cc:374] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:30.668732 2739 raft_consensus.cc:729] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e9f2ed18bd4467ea123dc19c92adcaf, State: Initialized, Role: FOLLOWER
I20260619 04:33:30.668937 2739 consensus_queue.cc:260] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:30.669363 2739 ts_tablet_manager.cc:1428] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 04:33:30.828699 2722 raft_consensus.cc:482] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:33:30.828917 2722 raft_consensus.cc:504] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:30.829526 2722 leader_election.cc:288] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dd08036edf54d5293ef71d27be76c4b (127.0.97.2:43405), 26e97c9f758a4e46a1afee4c07288303 (127.0.97.3:34065)
I20260619 04:33:30.830168 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "259dc80e72194fadbf560486b2911863" candidate_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26e97c9f758a4e46a1afee4c07288303" is_pre_election: true
I20260619 04:33:30.830200 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "259dc80e72194fadbf560486b2911863" candidate_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd08036edf54d5293ef71d27be76c4b" is_pre_election: true
I20260619 04:33:30.830435 2578 raft_consensus.cc:2454] T 259dc80e72194fadbf560486b2911863 P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e9f2ed18bd4467ea123dc19c92adcaf in term 0.
I20260619 04:33:30.830425 2440 raft_consensus.cc:2454] T 259dc80e72194fadbf560486b2911863 P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e9f2ed18bd4467ea123dc19c92adcaf in term 0.
I20260619 04:33:30.830870 2233 leader_election.cc:302] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26e97c9f758a4e46a1afee4c07288303, 8e9f2ed18bd4467ea123dc19c92adcaf; no voters:
I20260619 04:33:30.831182 2722 raft_consensus.cc:2781] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:33:30.831264 2722 raft_consensus.cc:482] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:33:30.831355 2722 raft_consensus.cc:3037] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:30.833290 2722 raft_consensus.cc:504] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:30.833823 2722 leader_election.cc:288] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf [CANDIDATE]: Term 1 election: Requested vote from peers 1dd08036edf54d5293ef71d27be76c4b (127.0.97.2:43405), 26e97c9f758a4e46a1afee4c07288303 (127.0.97.3:34065)
I20260619 04:33:30.834276 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "259dc80e72194fadbf560486b2911863" candidate_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26e97c9f758a4e46a1afee4c07288303"
I20260619 04:33:30.834331 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "259dc80e72194fadbf560486b2911863" candidate_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd08036edf54d5293ef71d27be76c4b"
I20260619 04:33:30.834514 2578 raft_consensus.cc:3037] T 259dc80e72194fadbf560486b2911863 P 26e97c9f758a4e46a1afee4c07288303 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:30.834550 2440 raft_consensus.cc:3037] T 259dc80e72194fadbf560486b2911863 P 1dd08036edf54d5293ef71d27be76c4b [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:30.837357 2578 raft_consensus.cc:2454] T 259dc80e72194fadbf560486b2911863 P 26e97c9f758a4e46a1afee4c07288303 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e9f2ed18bd4467ea123dc19c92adcaf in term 1.
I20260619 04:33:30.837522 2440 raft_consensus.cc:2454] T 259dc80e72194fadbf560486b2911863 P 1dd08036edf54d5293ef71d27be76c4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e9f2ed18bd4467ea123dc19c92adcaf in term 1.
I20260619 04:33:30.837819 2233 leader_election.cc:302] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26e97c9f758a4e46a1afee4c07288303, 8e9f2ed18bd4467ea123dc19c92adcaf; no voters:
I20260619 04:33:30.838182 2722 raft_consensus.cc:2781] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:30.838328 2722 raft_consensus.cc:686] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 LEADER]: Becoming Leader. State: Replica: 8e9f2ed18bd4467ea123dc19c92adcaf, State: Running, Role: LEADER
I20260619 04:33:30.838562 2722 consensus_queue.cc:237] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } }
I20260619 04:33:30.841161 2142 catalog_manager.cc:5495] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e9f2ed18bd4467ea123dc19c92adcaf (127.0.97.1). New cstate: current_term: 1 leader_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9f2ed18bd4467ea123dc19c92adcaf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } health_report { overall_health: UNKNOWN } } }
I20260619 04:33:30.863087 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:30.858026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.0.97.1:35787)
0619 04:33:30.858202 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:30.858209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:30.858566 (+ 357us) server_negotiation.cc:407] Connection header received
0619 04:33:30.858789 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:30.858793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:30.858855 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:30.858929 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:33:30.859360 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:30.860087 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:30.861450 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:30.861915 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:30.862020 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:30.862393 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:33:30.862504 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:33:30.862575 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:30.862735 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 04:33:30.862809 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 04:33:30.872283 2578 raft_consensus.cc:1261] T 259dc80e72194fadbf560486b2911863 P 26e97c9f758a4e46a1afee4c07288303 [term 1 FOLLOWER]: Refusing update from remote peer 8e9f2ed18bd4467ea123dc19c92adcaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:33:30.872400 2440 raft_consensus.cc:1261] T 259dc80e72194fadbf560486b2911863 P 1dd08036edf54d5293ef71d27be76c4b [term 1 FOLLOWER]: Refusing update from remote peer 8e9f2ed18bd4467ea123dc19c92adcaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:33:30.872918 2722 consensus_queue.cc:1035] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "26e97c9f758a4e46a1afee4c07288303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:33:30.873173 2721 consensus_queue.cc:1035] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dd08036edf54d5293ef71d27be76c4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 04:33:30.893929 2711 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 04:33:32.478866 2142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:33:32.479118 2142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:33:32.483428 2142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc27e1ee5edb40ae81b161e23a41c482: Sending DeleteTablet for 3 replicas of tablet 259dc80e72194fadbf560486b2911863
I20260619 04:33:32.484149 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet 259dc80e72194fadbf560486b2911863 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:32 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56914
I20260619 04:33:32.484319 2419 tablet_service.cc:1514] Processing DeleteTablet for tablet 259dc80e72194fadbf560486b2911863 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:32 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58760
I20260619 04:33:32.484843 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 2215
I20260619 04:33:32.484943 2559 tablet_service.cc:1514] Processing DeleteTablet for tablet 259dc80e72194fadbf560486b2911863 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:32 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53488
I20260619 04:33:32.497484 2794 tablet_replica.cc:330] T 259dc80e72194fadbf560486b2911863 P 1dd08036edf54d5293ef71d27be76c4b: stopping tablet replica
I20260619 04:33:32.497704 2794 raft_consensus.cc:2229] T 259dc80e72194fadbf560486b2911863 P 1dd08036edf54d5293ef71d27be76c4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:33:32.497936 2794 raft_consensus.cc:2258] T 259dc80e72194fadbf560486b2911863 P 1dd08036edf54d5293ef71d27be76c4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:32.499825 2793 tablet_replica.cc:330] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf: stopping tablet replica
I20260619 04:33:32.499994 2793 raft_consensus.cc:2229] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:33:32.500331 2793 raft_consensus.cc:2258] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:32.501437 2793 ts_tablet_manager.cc:1905] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:32.501854 2794 ts_tablet_manager.cc:1905] T 259dc80e72194fadbf560486b2911863 P 1dd08036edf54d5293ef71d27be76c4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:32.502628 2795 tablet_replica.cc:330] T 259dc80e72194fadbf560486b2911863 P 26e97c9f758a4e46a1afee4c07288303: stopping tablet replica
I20260619 04:33:32.502869 2795 raft_consensus.cc:2229] T 259dc80e72194fadbf560486b2911863 P 26e97c9f758a4e46a1afee4c07288303 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:33:32.503130 2795 raft_consensus.cc:2258] T 259dc80e72194fadbf560486b2911863 P 26e97c9f758a4e46a1afee4c07288303 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:32.504037 2795 ts_tablet_manager.cc:1905] T 259dc80e72194fadbf560486b2911863 P 26e97c9f758a4e46a1afee4c07288303: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:32.508636 2793 ts_tablet_manager.cc:1918] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 04:33:32.508757 2793 log.cc:1196] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-0/wal/wals/259dc80e72194fadbf560486b2911863
I20260619 04:33:32.509153 2793 ts_tablet_manager.cc:1939] T 259dc80e72194fadbf560486b2911863 P 8e9f2ed18bd4467ea123dc19c92adcaf: Deleting consensus metadata
I20260619 04:33:32.510200 2130 catalog_manager.cc:4841] TS 8e9f2ed18bd4467ea123dc19c92adcaf (127.0.97.1:35787): tablet 259dc80e72194fadbf560486b2911863 (table test-table [id=5a7d7fd9d9d644699ae2550d3a604119]) successfully deleted
I20260619 04:33:32.510274 2795 ts_tablet_manager.cc:1918] T 259dc80e72194fadbf560486b2911863 P 26e97c9f758a4e46a1afee4c07288303: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 04:33:32.510501 2795 log.cc:1196] T 259dc80e72194fadbf560486b2911863 P 26e97c9f758a4e46a1afee4c07288303: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/259dc80e72194fadbf560486b2911863
I20260619 04:33:32.510835 2795 ts_tablet_manager.cc:1939] T 259dc80e72194fadbf560486b2911863 P 26e97c9f758a4e46a1afee4c07288303: Deleting consensus metadata
I20260619 04:33:32.511672 2130 catalog_manager.cc:4841] TS 26e97c9f758a4e46a1afee4c07288303 (127.0.97.3:34065): tablet 259dc80e72194fadbf560486b2911863 (table test-table [id=5a7d7fd9d9d644699ae2550d3a604119]) successfully deleted
I20260619 04:33:32.512785 2794 ts_tablet_manager.cc:1918] T 259dc80e72194fadbf560486b2911863 P 1dd08036edf54d5293ef71d27be76c4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 04:33:32.512881 2794 log.cc:1196] T 259dc80e72194fadbf560486b2911863 P 1dd08036edf54d5293ef71d27be76c4b: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/259dc80e72194fadbf560486b2911863
I20260619 04:33:32.513159 2794 ts_tablet_manager.cc:1939] T 259dc80e72194fadbf560486b2911863 P 1dd08036edf54d5293ef71d27be76c4b: Deleting consensus metadata
I20260619 04:33:32.513883 2128 catalog_manager.cc:4841] TS 1dd08036edf54d5293ef71d27be76c4b (127.0.97.2:43405): tablet 259dc80e72194fadbf560486b2911863 (table test-table [id=5a7d7fd9d9d644699ae2550d3a604119]) successfully deleted
Jun 19 04:33:32 dist-test-slave-n3tq krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843610, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 04:33:32.521827 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:32.486628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56946 (local address 127.0.97.1:35787)
0619 04:33:32.501402 (+ 14774us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:32.501408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:32.501440 (+ 32us) server_negotiation.cc:407] Connection header received
0619 04:33:32.501524 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:32.501832 (+ 308us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:32.501989 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:32.502122 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:32.503593 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:32.504767 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:32.506438 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:32.507043 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:32.507162 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:32.517183 (+ 10021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:32.517199 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:32.517215 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:32.517252 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:32.519686 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:32.520259 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:32.520265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:32.520267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:32.520397 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:32.520715 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:32.520717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:32.520719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:32.521059 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:32.521274 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:32.521434 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 04:33:32.521584 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14639,"thread_start_us":14950,"threads_started":1}
W20260619 04:33:32.522830 2346 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56946
W20260619 04:33:32.523409 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 2215: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:33:32.523569 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 2215
I20260619 04:33:32.542070 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 2355
Jun 19 04:33:32 dist-test-slave-n3tq krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843610, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 04:33:32.555665 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:32.542738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.0.97.2:43405)
0619 04:33:32.543195 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:32.543199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:32.543213 (+ 14us) server_negotiation.cc:407] Connection header received
0619 04:33:32.543260 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:32.543263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:32.543331 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:32.543419 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:32.544567 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:32.545326 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:32.546484 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:32.546872 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:32.546984 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:32.550749 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:32.550762 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:32.550777 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:32.550809 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:32.553300 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:32.553857 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:32.553863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:32.553866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:32.553989 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:32.554305 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:32.554309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:32.554311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:32.554609 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:32.554865 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:32.555133 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 04:33:32.555346 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":231,"threads_started":1}
W20260619 04:33:32.556253 2485 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58774
W20260619 04:33:32.556854 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 2355: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:33:32.556994 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 2355
I20260619 04:33:32.574013 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 2494
Jun 19 04:33:32 dist-test-slave-n3tq krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843610, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 04:33:32.587620 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:32.574681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.0.97.3:34065)
0619 04:33:32.575119 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:32.575123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:32.575138 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:33:32.575199 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:32.575202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:32.575257 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:32.575341 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:32.576491 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:32.577228 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:32.578469 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:32.578876 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:32.578966 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:32.582881 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:32.582895 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:32.582912 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:32.582950 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:32.585369 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:32.585912 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:32.585918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:32.585921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:32.586052 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:32.586328 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:32.586330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:32.586333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:32.586681 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:32.586920 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:32.587092 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 04:33:32.587281 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":228,"threads_started":1}
W20260619 04:33:32.588224 2624 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53520
W20260619 04:33:32.588837 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 2494: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:33:32.588964 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 2494
I20260619 04:33:32.610524 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 2112
I20260619 04:33:32.611850 2184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:32.723091 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 2112
2026-06-19T04:33:32Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7883 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:33:32 dist-test-slave-n3tq krb5kdc[2803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:33:32 dist-test-slave-n3tq krb5kdc[2803](info): set up 2 sockets
Jun 19 04:33:32 dist-test-slave-n3tq krb5kdc[2803](info): commencing operation
krb5kdc: starting...
W20260619 04:33:33.716344 388 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:33:33 dist-test-slave-n3tq krb5kdc[2803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:33:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:33:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:33:33.877985 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46055
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:34.181039 2816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:34.181329 2816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:34.181445 2816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:34.188529 2816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:33:34.188642 2816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:34.188712 2816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:34.188791 2816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:33:34.188845 2816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:33:34.198570 2816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:34.199849 2816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:34.201167 2816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:34.212319 2824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:34.212297 2822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:34.212848 2816 server_base.cc:1034] running on GCE node
W20260619 04:33:34.212915 2821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:34.213702 2816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:34.215086 2816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:34.216325 2816 hybrid_clock.cc:648] HybridClock initialized: now 1781843614216268 us; error 37 us; skew 500 ppm
I20260619 04:33:34.216696 2816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:34 dist-test-slave-n3tq krb5kdc[2803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:34.219643 2816 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:33:34.221071 2816 webserver.cc:460] Webserver started at http://127.0.97.62:41177/ using document root <none> and password file <none>
I20260619 04:33:34.221719 2816 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:34.221819 2816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:34.222071 2816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:34.224990 2816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "29dbb120a61c4766af614ac64af63a18"
format_stamp: "Formatted at 2026-06-19 04:33:34 on dist-test-slave-n3tq"
server_key: "6dc8abb685f1d8d6a1f3de20acec70bb"
server_key_iv: "3a362919298e0c2044ebd6f8e3571a47"
server_key_version: "encryptionkey@0"
I20260619 04:33:34.225708 2816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "29dbb120a61c4766af614ac64af63a18"
format_stamp: "Formatted at 2026-06-19 04:33:34 on dist-test-slave-n3tq"
server_key: "6dc8abb685f1d8d6a1f3de20acec70bb"
server_key_iv: "3a362919298e0c2044ebd6f8e3571a47"
server_key_version: "encryptionkey@0"
I20260619 04:33:34.230792 2816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:33:34.234184 2831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:34.235512 2816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:33:34.235687 2816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "29dbb120a61c4766af614ac64af63a18"
format_stamp: "Formatted at 2026-06-19 04:33:34 on dist-test-slave-n3tq"
server_key: "6dc8abb685f1d8d6a1f3de20acec70bb"
server_key_iv: "3a362919298e0c2044ebd6f8e3571a47"
server_key_version: "encryptionkey@0"
I20260619 04:33:34.235898 2816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:34.270032 2816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:34.274102 2816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:34.274371 2816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:34.290787 2816 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46055
I20260619 04:33:34.290820 2882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46055 every 8 connection(s)
I20260619 04:33:34.292083 2816 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:33:34.293558 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 2816
I20260619 04:33:34.293727 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:33:34.294307 388 external_mini_cluster.cc:1428] Setting key 47e2819cafdbf2fc8bd9f40a86c65a91
I20260619 04:33:34.296123 2883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 04:33:34 dist-test-slave-n3tq krb5kdc[2803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:34.308646 2883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18: Bootstrap starting.
I20260619 04:33:34.312587 2883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:34.313731 2883 log.cc:824] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:34.316754 2886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:34.296124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44708 (local address 127.0.97.62:46055)
0619 04:33:34.296856 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:34.296868 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:34.296908 (+ 40us) server_negotiation.cc:407] Connection header received
0619 04:33:34.298456 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:34.298476 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:34.298888 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:34.299218 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:34.300935 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:34.302252 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:34.303865 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:34.304559 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:34.304712 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:34.308983 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:34.308994 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:34.309013 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:34.309055 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:34.312500 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:34.313149 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:34.313157 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:34.313165 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:34.313363 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:34.313667 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:34.313671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:34.313674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:34.314359 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:34.314584 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:34.315026 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 04:33:34.315428 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":313,"threads_started":1}
I20260619 04:33:34.317037 2883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18: No bootstrap required, opened a new log
I20260619 04:33:34.324021 2883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dbb120a61c4766af614ac64af63a18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46055 } }
I20260619 04:33:34.324365 2883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:34.324481 2883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29dbb120a61c4766af614ac64af63a18, State: Initialized, Role: FOLLOWER
I20260619 04:33:34.325037 2883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dbb120a61c4766af614ac64af63a18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46055 } }
I20260619 04:33:34.325239 2883 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:33:34.325356 2883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:33:34.325495 2883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:34.327453 2883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dbb120a61c4766af614ac64af63a18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46055 } }
I20260619 04:33:34.327906 2883 leader_election.cc:302] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29dbb120a61c4766af614ac64af63a18; no voters:
I20260619 04:33:34.328418 2883 leader_election.cc:288] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:33:34.328572 2888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:34.329689 2888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18 [term 1 LEADER]: Becoming Leader. State: Replica: 29dbb120a61c4766af614ac64af63a18, State: Running, Role: LEADER
I20260619 04:33:34.330127 2888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dbb120a61c4766af614ac64af63a18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46055 } }
I20260619 04:33:34.330508 2883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:33:34.332644 2890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29dbb120a61c4766af614ac64af63a18. Latest consensus state: current_term: 1 leader_uuid: "29dbb120a61c4766af614ac64af63a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dbb120a61c4766af614ac64af63a18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46055 } } }
I20260619 04:33:34.332712 2889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29dbb120a61c4766af614ac64af63a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dbb120a61c4766af614ac64af63a18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46055 } } }
I20260619 04:33:34.332947 2890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18 [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:34.333043 2889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18 [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:34.333734 2897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:33:34.338737 2897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:33:34.347674 2897 catalog_manager.cc:1347] Generated new cluster ID: 8ac79d16e7334f4194a47d70b0bbd76d
I20260619 04:33:34.347780 2897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:33:34.368950 2897 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:33:34.370177 2897 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:33:34.386277 2897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29dbb120a61c4766af614ac64af63a18: Generated new TSK 0
I20260619 04:33:34.387063 2897 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 04:33:34.433351 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46055
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:34.741067 2909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:34.741353 2909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:34.741492 2909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:34.748571 2909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:34.748688 2909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:34.748850 2909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:33:34.758420 2909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:34.759699 2909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:34.761021 2909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:34.773079 2915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:34.773135 2917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:34.773159 2914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:34.774152 2909 server_base.cc:1034] running on GCE node
I20260619 04:33:34.774696 2909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:34.775590 2909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:34.776775 2909 hybrid_clock.cc:648] HybridClock initialized: now 1781843614776725 us; error 47 us; skew 500 ppm
I20260619 04:33:34.777064 2909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:34 dist-test-slave-n3tq krb5kdc[2803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:34.779981 2909 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:33:34.781311 2909 webserver.cc:460] Webserver started at http://127.0.97.1:36041/ using document root <none> and password file <none>
I20260619 04:33:34.781958 2909 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:34.782068 2909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:34.782356 2909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:34.785161 2909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "cbc0595b3e0c4eecadb251ef2f759641"
format_stamp: "Formatted at 2026-06-19 04:33:34 on dist-test-slave-n3tq"
server_key: "7922c40838508336e2dd85c5eb142e4f"
server_key_iv: "4fcb1a6dd8ef346c8d9ef498311d5584"
server_key_version: "encryptionkey@0"
I20260619 04:33:34.785895 2909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cbc0595b3e0c4eecadb251ef2f759641"
format_stamp: "Formatted at 2026-06-19 04:33:34 on dist-test-slave-n3tq"
server_key: "7922c40838508336e2dd85c5eb142e4f"
server_key_iv: "4fcb1a6dd8ef346c8d9ef498311d5584"
server_key_version: "encryptionkey@0"
I20260619 04:33:34.790607 2909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 04:33:34.793861 2924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:34.795070 2909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 04:33:34.795213 2909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "cbc0595b3e0c4eecadb251ef2f759641"
format_stamp: "Formatted at 2026-06-19 04:33:34 on dist-test-slave-n3tq"
server_key: "7922c40838508336e2dd85c5eb142e4f"
server_key_iv: "4fcb1a6dd8ef346c8d9ef498311d5584"
server_key_version: "encryptionkey@0"
I20260619 04:33:34.795394 2909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:34.816967 2909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:34.821144 2909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:34.821414 2909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:34.822336 2909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:34.823685 2909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:34.823763 2909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:34.823884 2909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:34.823935 2909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:34.852861 2909 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33221
I20260619 04:33:34.852907 3036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33221 every 8 connection(s)
I20260619 04:33:34.854106 2909 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:33:34.860875 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 2909
I20260619 04:33:34.861053 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:33:34.861606 388 external_mini_cluster.cc:1428] Setting key 5308ee22127aa91cc8f7afefc13e0465
Jun 19 04:33:34 dist-test-slave-n3tq krb5kdc[2803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:34.872849 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:34.856538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54589 (local address 127.0.97.62:46055)
0619 04:33:34.856937 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:34.856942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:34.857901 (+ 959us) server_negotiation.cc:407] Connection header received
0619 04:33:34.859646 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:34.859651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:34.859734 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:34.859855 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:34.861412 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:34.862132 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:34.863846 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:34.864234 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:34.864336 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:34.867674 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:34.867679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:34.867682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:34.867712 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:34.869995 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:34.870660 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:34.870664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:34.870667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:34.870778 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:34.871165 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:34.871168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:34.871170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:34.871443 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:34.871574 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:34.872247 (+ 673us) server_negotiation.cc:299] Negotiation successful
0619 04:33:34.872505 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":190,"threads_started":1}
I20260619 04:33:34.873770 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:34.857089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46055 (local address 127.0.97.1:54589)
0619 04:33:34.857755 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:34.857791 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:34.859391 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:34.860090 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:34.860098 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:34.860664 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:34.861254 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:34.861273 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:34.862273 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:34.862277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:34.863712 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:34.863724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:34.864439 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:34.864445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:34.864686 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:34.865290 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:34.865319 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:34.867466 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:34.870123 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:34.870132 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:34.870150 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:34.870544 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:34.870874 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:34.870880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:34.870883 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:34.871057 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:34.871561 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:34.871568 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:34.871909 (+ 341us) client_negotiation.cc:769] Sending connection context
0619 04:33:34.872138 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 04:33:34.872676 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":249,"threads_started":1}
I20260619 04:33:34.875231 3037 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46055
I20260619 04:33:34.875583 3037 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:34.876435 3037 heartbeater.cc:511] Master 127.0.97.62:46055 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 04:33:34.878829 2848 ts_manager.cc:194] Registered new tserver with Master: cbc0595b3e0c4eecadb251ef2f759641 (127.0.97.1:33221)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 04:33:34.880827 2848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 04:33:34.899260 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46055
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:35.202627 3043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:35.202875 3043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:35.202960 3043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:35.209873 3043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:35.209967 3043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:35.210069 3043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:33:35.221611 3043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:35.223212 3043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:35.225013 3043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:35.238628 3048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:35.238615 3049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:35.239202 3051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:35.240545 3043 server_base.cc:1034] running on GCE node
I20260619 04:33:35.241267 3043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:35.242441 3043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:35.243649 3043 hybrid_clock.cc:648] HybridClock initialized: now 1781843615243603 us; error 42 us; skew 500 ppm
I20260619 04:33:35.243942 3043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:35 dist-test-slave-n3tq krb5kdc[2803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:35.247468 3043 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:33:35.248878 3043 webserver.cc:460] Webserver started at http://127.0.97.2:46459/ using document root <none> and password file <none>
I20260619 04:33:35.249728 3043 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:35.249856 3043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:35.250180 3043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:35.253852 3043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "a15c12a2f36343758a3b6b67a9730c67"
format_stamp: "Formatted at 2026-06-19 04:33:35 on dist-test-slave-n3tq"
server_key: "3c8a519a857b026c437aafad59c1d438"
server_key_iv: "e4d99610e212ca7f0298ca4bc1ee0758"
server_key_version: "encryptionkey@0"
I20260619 04:33:35.254783 3043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a15c12a2f36343758a3b6b67a9730c67"
format_stamp: "Formatted at 2026-06-19 04:33:35 on dist-test-slave-n3tq"
server_key: "3c8a519a857b026c437aafad59c1d438"
server_key_iv: "e4d99610e212ca7f0298ca4bc1ee0758"
server_key_version: "encryptionkey@0"
I20260619 04:33:35.261263 3043 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 04:33:35.264665 3058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:35.265921 3043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:33:35.266064 3043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "a15c12a2f36343758a3b6b67a9730c67"
format_stamp: "Formatted at 2026-06-19 04:33:35 on dist-test-slave-n3tq"
server_key: "3c8a519a857b026c437aafad59c1d438"
server_key_iv: "e4d99610e212ca7f0298ca4bc1ee0758"
server_key_version: "encryptionkey@0"
I20260619 04:33:35.266244 3043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:35.284407 3043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:35.288342 3043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:35.288616 3043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:35.289553 3043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:35.290889 3043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:35.290961 3043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:35.291059 3043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:35.291105 3043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:35.320812 3043 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39569
I20260619 04:33:35.320849 3170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39569 every 8 connection(s)
I20260619 04:33:35.321998 3043 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:33:35.324407 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 3043
I20260619 04:33:35.324604 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:33:35.325124 388 external_mini_cluster.cc:1428] Setting key 16a07bb0af5128466950858773ebfe12
Jun 19 04:33:35 dist-test-slave-n3tq krb5kdc[2803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 04:33:35.341679 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:35.324334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42183 (local address 127.0.97.62:46055)
0619 04:33:35.324516 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:35.324522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:35.325963 (+ 1441us) server_negotiation.cc:407] Connection header received
0619 04:33:35.328049 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:35.328054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:35.328131 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:35.328217 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:35.329704 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:35.330414 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:35.331974 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:35.332396 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:35.332532 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:35.335967 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:35.335972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:35.335975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:35.336005 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:35.338870 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:35.339524 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:35.339530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:35.339533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:35.339705 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:35.340071 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:35.340076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:35.340079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:35.340482 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:35.340629 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:35.341274 (+ 645us) server_negotiation.cc:299] Negotiation successful
0619 04:33:35.341439 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 04:33:35.342653 3173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:35.325031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46055 (local address 127.0.97.2:42183)
0619 04:33:35.325799 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:35.325856 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:35.327791 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:35.328478 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:35.328491 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:35.329007 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:35.329550 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:35.329567 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:35.330552 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:35.330556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:35.331818 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:35.331830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:35.332615 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:35.332622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:35.332841 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:35.333470 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:35.333495 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:35.335780 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:35.338996 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:35.339003 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:35.339032 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:35.339423 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:35.339800 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:35.339804 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:35.339806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:35.339971 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:35.340593 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:35.340599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:35.340922 (+ 323us) client_negotiation.cc:769] Sending connection context
0619 04:33:35.341175 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 04:33:35.341541 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":295,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 04:33:35.344013 3171 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46055
I20260619 04:33:35.344336 3171 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:35.345144 3171 heartbeater.cc:511] Master 127.0.97.62:46055 requested a full tablet report, sending...
I20260619 04:33:35.346521 2848 ts_manager.cc:194] Registered new tserver with Master: a15c12a2f36343758a3b6b67a9730c67 (127.0.97.2:39569)
I20260619 04:33:35.347280 2848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 04:33:35.361667 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46055
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:35.659713 3176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:35.659978 3176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:35.660074 3176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:35.667109 3176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:35.667218 3176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:35.667328 3176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:33:35.676831 3176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:35.678027 3176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:35.679224 3176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:35.691356 3184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:35.691319 3182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:35.691399 3181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:35.692117 3176 server_base.cc:1034] running on GCE node
I20260619 04:33:35.692715 3176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:35.693524 3176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:35.694717 3176 hybrid_clock.cc:648] HybridClock initialized: now 1781843615694660 us; error 37 us; skew 500 ppm
I20260619 04:33:35.694993 3176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:35 dist-test-slave-n3tq krb5kdc[2803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:35.697824 3176 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:33:35.699208 3176 webserver.cc:460] Webserver started at http://127.0.97.3:42487/ using document root <none> and password file <none>
I20260619 04:33:35.699842 3176 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:35.699950 3176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:35.700229 3176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:35.702993 3176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "d4edc1b671454479a459446bc0ec2dd4"
format_stamp: "Formatted at 2026-06-19 04:33:35 on dist-test-slave-n3tq"
server_key: "2416bbb8d74b85eb91da71ecd3c0c3c5"
server_key_iv: "44891428d8e5f285ae5ef24f2368b983"
server_key_version: "encryptionkey@0"
I20260619 04:33:35.703715 3176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d4edc1b671454479a459446bc0ec2dd4"
format_stamp: "Formatted at 2026-06-19 04:33:35 on dist-test-slave-n3tq"
server_key: "2416bbb8d74b85eb91da71ecd3c0c3c5"
server_key_iv: "44891428d8e5f285ae5ef24f2368b983"
server_key_version: "encryptionkey@0"
I20260619 04:33:35.708348 3176 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260619 04:33:35.711457 3191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:35.712673 3176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 04:33:35.712812 3176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "d4edc1b671454479a459446bc0ec2dd4"
format_stamp: "Formatted at 2026-06-19 04:33:35 on dist-test-slave-n3tq"
server_key: "2416bbb8d74b85eb91da71ecd3c0c3c5"
server_key_iv: "44891428d8e5f285ae5ef24f2368b983"
server_key_version: "encryptionkey@0"
I20260619 04:33:35.712983 3176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:35.751636 3176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:35.755767 3176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:35.756034 3176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:35.757011 3176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:35.758291 3176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:35.758365 3176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:35.758481 3176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:35.758533 3176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:35.786895 3176 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34515
I20260619 04:33:35.786950 3303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34515 every 8 connection(s)
I20260619 04:33:35.788077 3176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:33:35.796247 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 3176
I20260619 04:33:35.796403 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:33:35.796806 388 external_mini_cluster.cc:1428] Setting key 0e3c9192fd61afc1bbf05bc6f9eae9ef
Jun 19 04:33:35 dist-test-slave-n3tq krb5kdc[2803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:35.806250 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:35.790412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51123 (local address 127.0.97.62:46055)
0619 04:33:35.790593 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:35.790597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:35.791666 (+ 1069us) server_negotiation.cc:407] Connection header received
0619 04:33:35.793427 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:35.793432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:35.793508 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:35.793591 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:35.795001 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:35.795725 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:35.797300 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:35.797755 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:35.797868 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:35.801292 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:35.801297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:35.801300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:35.801330 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:35.803701 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:35.804362 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:35.804366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:35.804368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:35.804532 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:35.804878 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:35.804882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:35.804885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:35.805152 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:35.805255 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:35.805848 (+ 593us) server_negotiation.cc:299] Negotiation successful
0619 04:33:35.806001 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 04:33:35.807173 3306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:35.790850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46055 (local address 127.0.97.3:51123)
0619 04:33:35.791493 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:35.791533 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:35.793169 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:35.793822 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:35.793830 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:35.794324 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:35.794849 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:35.794865 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:35.795863 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:35.795867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:35.797176 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:35.797188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:35.797945 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:35.797952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:35.798178 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:35.798838 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:35.798866 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:35.801115 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:35.803829 (+ 2714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:35.803835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:35.803849 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:35.804237 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:35.804622 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:35.804625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:35.804627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:35.804793 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:35.805242 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:35.805247 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:35.805547 (+ 300us) client_negotiation.cc:769] Sending connection context
0619 04:33:35.805755 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 04:33:35.806149 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":257,"threads_started":1}
I20260619 04:33:35.808679 3304 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46055
I20260619 04:33:35.808954 3304 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:35.809630 3304 heartbeater.cc:511] Master 127.0.97.62:46055 requested a full tablet report, sending...
I20260619 04:33:35.811115 2848 ts_manager.cc:194] Registered new tserver with Master: d4edc1b671454479a459446bc0ec2dd4 (127.0.97.3:34515)
I20260619 04:33:35.811839 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:33:35.812150 2848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51123
W20260619 04:33:35.837827 3039 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:33:35.835567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44724 (local address 127.0.97.62:46055)
0619 04:33:35.835772 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:35.835777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:35.836193 (+ 416us) server_negotiation.cc:407] Connection header received
0619 04:33:35.836321 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:35.836324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:35.836386 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:35.836559 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:35.837622 (+ 1063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44724: BlockingRecv error: recv got EOF from 127.0.0.1:44724 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 04:33:35.841097 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 2909
W20260619 04:33:35.843376 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 2909: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:33221: server requires authentication, but client does not have Kerberos credentials available
I20260619 04:33:35.843537 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 2909
I20260619 04:33:35.856024 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 3043
W20260619 04:33:35.858417 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 3043: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:39569: server requires authentication, but client does not have Kerberos credentials available
I20260619 04:33:35.858560 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 3043
I20260619 04:33:35.870182 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 3176
W20260619 04:33:35.872793 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 3176: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:34515: server requires authentication, but client does not have Kerberos credentials available
I20260619 04:33:35.872938 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 3176
I20260619 04:33:35.885322 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 2816
I20260619 04:33:35.886411 2878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:35.985929 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 2816
2026-06-19T04:33:35Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3260 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:33:36 dist-test-slave-n3tq krb5kdc[3321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:33:36 dist-test-slave-n3tq krb5kdc[3321](info): set up 2 sockets
Jun 19 04:33:36 dist-test-slave-n3tq krb5kdc[3321](info): commencing operation
krb5kdc: starting...
W20260619 04:33:36.976524 388 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:33:37 dist-test-slave-n3tq krb5kdc[3321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:33:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:33:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:33:37.146171 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40677
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:37.473482 3334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:37.473779 3334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:37.473899 3334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:37.480901 3334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:33:37.481024 3334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:37.481091 3334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:37.481163 3334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:33:37.481228 3334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:33:37.490634 3334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:37.491909 3334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:37.493178 3334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:37.503895 3339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:37.503895 3342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:37.503927 3340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:37.504438 3334 server_base.cc:1034] running on GCE node
I20260619 04:33:37.505266 3334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:37.506494 3334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:37.507678 3334 hybrid_clock.cc:648] HybridClock initialized: now 1781843617507632 us; error 38 us; skew 500 ppm
I20260619 04:33:37.507965 3334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:37 dist-test-slave-n3tq krb5kdc[3321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:37.511049 3334 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:33:37.512501 3334 webserver.cc:460] Webserver started at http://127.0.97.62:37385/ using document root <none> and password file <none>
I20260619 04:33:37.513168 3334 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:37.513279 3334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:37.513561 3334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:37.516366 3334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "5ad1196d420a4f109b07faf100723ec1"
format_stamp: "Formatted at 2026-06-19 04:33:37 on dist-test-slave-n3tq"
server_key: "72ffe4276393a6d7aa45e637264361c7"
server_key_iv: "199d5567ad6aada3d58888f2204704e3"
server_key_version: "encryptionkey@0"
I20260619 04:33:37.517141 3334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "5ad1196d420a4f109b07faf100723ec1"
format_stamp: "Formatted at 2026-06-19 04:33:37 on dist-test-slave-n3tq"
server_key: "72ffe4276393a6d7aa45e637264361c7"
server_key_iv: "199d5567ad6aada3d58888f2204704e3"
server_key_version: "encryptionkey@0"
I20260619 04:33:37.521929 3334 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 04:33:37.525111 3349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:37.526311 3334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:33:37.526463 3334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "5ad1196d420a4f109b07faf100723ec1"
format_stamp: "Formatted at 2026-06-19 04:33:37 on dist-test-slave-n3tq"
server_key: "72ffe4276393a6d7aa45e637264361c7"
server_key_iv: "199d5567ad6aada3d58888f2204704e3"
server_key_version: "encryptionkey@0"
I20260619 04:33:37.526623 3334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:37.553309 3334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:37.557446 3334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:37.557693 3334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:37.573542 3334 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40677
I20260619 04:33:37.573575 3400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40677 every 8 connection(s)
I20260619 04:33:37.574815 3334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:33:37.578339 3401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:37.584968 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 3334
I20260619 04:33:37.585186 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:33:37.585909 388 external_mini_cluster.cc:1428] Setting key 58d5ce0d49b98cfd806fcc1d0c694bed
I20260619 04:33:37.588832 3401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1: Bootstrap starting.
I20260619 04:33:37.592115 3401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:37.593216 3401 log.cc:824] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:37.596064 3401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1: No bootstrap required, opened a new log
Jun 19 04:33:37 dist-test-slave-n3tq krb5kdc[3321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:37.605787 3401 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1196d420a4f109b07faf100723ec1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40677 } }
I20260619 04:33:37.606251 3401 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:37.606374 3401 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad1196d420a4f109b07faf100723ec1, State: Initialized, Role: FOLLOWER
I20260619 04:33:37.607054 3401 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1196d420a4f109b07faf100723ec1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40677 } }
I20260619 04:33:37.607328 3401 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:33:37.607462 3401 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:33:37.607626 3401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:37.608727 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:37.587658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41362 (local address 127.0.97.62:40677)
0619 04:33:37.588337 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:37.588349 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:37.588389 (+ 40us) server_negotiation.cc:407] Connection header received
0619 04:33:37.590070 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:37.590096 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:37.590483 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:37.590797 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:37.592492 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:37.593860 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:37.595493 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:37.596155 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:37.596303 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:37.600479 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:37.600492 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:37.600513 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:37.600559 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:37.604305 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:37.604948 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:37.604957 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:37.604967 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:37.605167 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:37.605479 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:37.605484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:37.605487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:37.606222 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:37.606499 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:37.606899 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 04:33:37.607334 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":305,"threads_started":1}
I20260619 04:33:37.610374 3401 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1196d420a4f109b07faf100723ec1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40677 } }
I20260619 04:33:37.610857 3401 leader_election.cc:302] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ad1196d420a4f109b07faf100723ec1; no voters:
I20260619 04:33:37.611393 3401 leader_election.cc:288] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:33:37.611536 3406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:37.612622 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:33:37.613359 3401 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:33:37.618394 3406 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1 [term 1 LEADER]: Becoming Leader. State: Replica: 5ad1196d420a4f109b07faf100723ec1, State: Running, Role: LEADER
I20260619 04:33:37.619002 3406 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1196d420a4f109b07faf100723ec1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40677 } }
I20260619 04:33:37.622044 3407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ad1196d420a4f109b07faf100723ec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1196d420a4f109b07faf100723ec1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40677 } } }
I20260619 04:33:37.622421 3407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1 [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:37.622851 3408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ad1196d420a4f109b07faf100723ec1. Latest consensus state: current_term: 1 leader_uuid: "5ad1196d420a4f109b07faf100723ec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1196d420a4f109b07faf100723ec1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40677 } } }
I20260619 04:33:37.623085 3408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1 [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:37.631076 3421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:33:37.636183 3424 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 04:33:37.641103 3421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:33:37.650673 3421 catalog_manager.cc:1347] Generated new cluster ID: 23fa75c1fbd44ad09fe9d9c65870bcb6
I20260619 04:33:37.650894 3421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:33:37.659132 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:37.650433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41372 (local address 127.0.97.62:40677)
0619 04:33:37.650582 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:37.650587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:37.651282 (+ 695us) server_negotiation.cc:407] Connection header received
0619 04:33:37.651466 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:37.651472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:37.651554 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:37.651673 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:37.652818 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:37.653934 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:37.655666 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:37.656309 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:37.656438 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:37.657526 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:37.657533 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:37.657538 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:33:37.658336 (+ 798us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:37.658408 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:33:37.658445 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:37.658543 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:37.658662 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 04:33:37.658755 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 04:33:37.683599 3421 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:33:37.685482 3421 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:33:37.695400 3421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ad1196d420a4f109b07faf100723ec1: Generated new TSK 0
I20260619 04:33:37.696291 3421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:33:37.710049 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 3334
I20260619 04:33:37.711215 3396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:37.801048 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 3334
2026-06-19T04:33:37Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1810 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:33:37 dist-test-slave-n3tq krb5kdc[3436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:33:37 dist-test-slave-n3tq krb5kdc[3436](info): set up 2 sockets
Jun 19 04:33:37 dist-test-slave-n3tq krb5kdc[3436](info): commencing operation
krb5kdc: starting...
W20260619 04:33:38.788558 388 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:33:38 dist-test-slave-n3tq krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:33:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:33:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:33:38.959978 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44875
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:39.266505 3449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:39.266876 3449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:39.267005 3449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:39.274047 3449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:33:39.274149 3449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:39.274219 3449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:39.274292 3449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:33:39.274354 3449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:33:39.284036 3449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:39.285343 3449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:39.286594 3449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:39.297477 3454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:39.297415 3455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:39.297454 3457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:39.298457 3449 server_base.cc:1034] running on GCE node
I20260619 04:33:39.299252 3449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:39.300678 3449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:39.301878 3449 hybrid_clock.cc:648] HybridClock initialized: now 1781843619301829 us; error 40 us; skew 500 ppm
I20260619 04:33:39.302189 3449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:39 dist-test-slave-n3tq krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:39.305080 3449 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:33:39.306505 3449 webserver.cc:460] Webserver started at http://127.0.97.62:38095/ using document root <none> and password file <none>
I20260619 04:33:39.307157 3449 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:39.307276 3449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:39.307544 3449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:39.310410 3449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "03c2d05cb4d34a69abeca102e5b5a024"
format_stamp: "Formatted at 2026-06-19 04:33:39 on dist-test-slave-n3tq"
server_key: "b62edca97c6445ca3deef7304050c3a1"
server_key_iv: "4ae2e05b0bff5adc650e0e811c3713ed"
server_key_version: "encryptionkey@0"
I20260619 04:33:39.311124 3449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "03c2d05cb4d34a69abeca102e5b5a024"
format_stamp: "Formatted at 2026-06-19 04:33:39 on dist-test-slave-n3tq"
server_key: "b62edca97c6445ca3deef7304050c3a1"
server_key_iv: "4ae2e05b0bff5adc650e0e811c3713ed"
server_key_version: "encryptionkey@0"
I20260619 04:33:39.316309 3449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 04:33:39.319592 3464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:39.321105 3449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:33:39.321286 3449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "03c2d05cb4d34a69abeca102e5b5a024"
format_stamp: "Formatted at 2026-06-19 04:33:39 on dist-test-slave-n3tq"
server_key: "b62edca97c6445ca3deef7304050c3a1"
server_key_iv: "4ae2e05b0bff5adc650e0e811c3713ed"
server_key_version: "encryptionkey@0"
I20260619 04:33:39.321470 3449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:39.342676 3449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:39.346756 3449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:39.347035 3449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:39.363425 3449 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44875
I20260619 04:33:39.363445 3515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44875 every 8 connection(s)
I20260619 04:33:39.364830 3449 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:33:39.366389 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 3449
I20260619 04:33:39.366631 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:33:39.367394 388 external_mini_cluster.cc:1428] Setting key 9c04f683564e6fe017c4dd1a6a7ae98b
I20260619 04:33:39.369274 3516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 04:33:39 dist-test-slave-n3tq krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:39.379902 3516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024: Bootstrap starting.
I20260619 04:33:39.383285 3516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:39.384410 3516 log.cc:824] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:39.387579 3516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024: No bootstrap required, opened a new log
I20260619 04:33:39.388078 3519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:39.369391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53604 (local address 127.0.97.62:44875)
0619 04:33:39.370107 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:39.370119 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:39.370157 (+ 38us) server_negotiation.cc:407] Connection header received
0619 04:33:39.371654 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:39.371676 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:39.372084 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:39.372490 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:39.373828 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:39.375123 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:39.376303 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:39.377039 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:39.377215 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:39.381101 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:39.381109 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:39.381125 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:39.381167 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:39.384009 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:39.384881 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:39.384888 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:39.384895 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:39.385114 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:39.385480 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:39.385488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:39.385491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:39.386076 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:39.386288 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:39.386716 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 04:33:39.387074 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":315,"threads_started":1}
I20260619 04:33:39.394804 3516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c2d05cb4d34a69abeca102e5b5a024" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44875 } }
I20260619 04:33:39.395161 3516 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:39.395260 3516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03c2d05cb4d34a69abeca102e5b5a024, State: Initialized, Role: FOLLOWER
I20260619 04:33:39.395797 3516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c2d05cb4d34a69abeca102e5b5a024" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44875 } }
I20260619 04:33:39.395998 3516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:33:39.396121 3516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:33:39.396272 3516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:39.398609 3516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c2d05cb4d34a69abeca102e5b5a024" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44875 } }
I20260619 04:33:39.399076 3516 leader_election.cc:302] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03c2d05cb4d34a69abeca102e5b5a024; no voters:
I20260619 04:33:39.399600 3516 leader_election.cc:288] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:33:39.399709 3521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:39.400326 3521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024 [term 1 LEADER]: Becoming Leader. State: Replica: 03c2d05cb4d34a69abeca102e5b5a024, State: Running, Role: LEADER
I20260619 04:33:39.400861 3521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c2d05cb4d34a69abeca102e5b5a024" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44875 } }
I20260619 04:33:39.401518 3516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:33:39.403576 3522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03c2d05cb4d34a69abeca102e5b5a024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c2d05cb4d34a69abeca102e5b5a024" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44875 } } }
I20260619 04:33:39.403573 3523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03c2d05cb4d34a69abeca102e5b5a024. Latest consensus state: current_term: 1 leader_uuid: "03c2d05cb4d34a69abeca102e5b5a024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c2d05cb4d34a69abeca102e5b5a024" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44875 } } }
I20260619 04:33:39.403990 3522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024 [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:39.404089 3523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024 [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:39.405551 3526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:33:39.409970 3526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:33:39.418512 3526 catalog_manager.cc:1347] Generated new cluster ID: d3e906a1a9ec4908a83d97dbc83bfc0d
I20260619 04:33:39.418613 3526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:33:39.441838 3526 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:33:39.443156 3526 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:33:39.451982 3526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03c2d05cb4d34a69abeca102e5b5a024: Generated new TSK 0
I20260619 04:33:39.452872 3526 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 04:33:39.502071 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44875
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:39.816905 3542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:39.817162 3542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:39.817258 3542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:39.824301 3542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:39.824411 3542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:39.824572 3542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:33:39.834141 3542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:39.835363 3542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:39.836658 3542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:39.848521 3547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:39.848426 3548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:39.848730 3550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:39.849661 3542 server_base.cc:1034] running on GCE node
I20260619 04:33:39.850229 3542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:39.851104 3542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:39.852288 3542 hybrid_clock.cc:648] HybridClock initialized: now 1781843619852247 us; error 35 us; skew 500 ppm
I20260619 04:33:39.852600 3542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:39 dist-test-slave-n3tq krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:39.855481 3542 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:33:39.856885 3542 webserver.cc:460] Webserver started at http://127.0.97.1:36193/ using document root <none> and password file <none>
I20260619 04:33:39.857511 3542 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:39.857611 3542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:39.857888 3542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:39.860647 3542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "87b14bc5a1fd435d838efd07db80d4d1"
format_stamp: "Formatted at 2026-06-19 04:33:39 on dist-test-slave-n3tq"
server_key: "49da5bdc19cd6d4ee6c9100c102e279f"
server_key_iv: "347e15bb1d81ebe74ebfafe5d05f7168"
server_key_version: "encryptionkey@0"
I20260619 04:33:39.861383 3542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "87b14bc5a1fd435d838efd07db80d4d1"
format_stamp: "Formatted at 2026-06-19 04:33:39 on dist-test-slave-n3tq"
server_key: "49da5bdc19cd6d4ee6c9100c102e279f"
server_key_iv: "347e15bb1d81ebe74ebfafe5d05f7168"
server_key_version: "encryptionkey@0"
I20260619 04:33:39.866161 3542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 04:33:39.869356 3557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:39.870577 3542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260619 04:33:39.870715 3542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "87b14bc5a1fd435d838efd07db80d4d1"
format_stamp: "Formatted at 2026-06-19 04:33:39 on dist-test-slave-n3tq"
server_key: "49da5bdc19cd6d4ee6c9100c102e279f"
server_key_iv: "347e15bb1d81ebe74ebfafe5d05f7168"
server_key_version: "encryptionkey@0"
I20260619 04:33:39.870874 3542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:39.893683 3542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:39.898113 3542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:39.898370 3542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:39.899232 3542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:39.900496 3542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:39.900570 3542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:39.900682 3542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:39.900743 3542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:39.928174 3542 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40187
I20260619 04:33:39.928221 3669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40187 every 8 connection(s)
I20260619 04:33:39.929399 3542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:33:39.930568 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 3542
I20260619 04:33:39.930764 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:33:39.931432 388 external_mini_cluster.cc:1428] Setting key 63f071f633e74764cce33a263a040db5
Jun 19 04:33:39 dist-test-slave-n3tq krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 04:33:39.949213 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:39.931731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46641 (local address 127.0.97.62:44875)
0619 04:33:39.932296 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:39.932302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:39.935041 (+ 2739us) server_negotiation.cc:407] Connection header received
0619 04:33:39.935088 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:39.935092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:39.935148 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:39.935224 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:39.937167 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:39.937873 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:39.939524 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:39.939904 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:39.940003 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:39.943544 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:39.943550 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:39.943553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:39.943582 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:39.946385 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:39.947067 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:39.947073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:39.947077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:39.947186 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:39.947577 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:39.947583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:39.947587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:39.947840 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:39.947972 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:39.948643 (+ 671us) server_negotiation.cc:299] Negotiation successful
0619 04:33:39.948850 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":337,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 04:33:39.950300 3673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:39.932400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44875 (local address 127.0.97.1:46641)
0619 04:33:39.933135 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:39.933184 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:39.934772 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:39.935510 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:39.935522 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:39.936250 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:39.937017 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:39.937041 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:39.938022 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:39.938027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:39.939398 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:39.939412 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:39.940106 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:39.940113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:39.940376 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:39.941028 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:39.941052 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:39.943332 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:39.946519 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:39.946529 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:39.946546 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:39.946949 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:39.947281 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:39.947286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:39.947290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:39.947470 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:39.947956 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:39.947964 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:39.948286 (+ 322us) client_negotiation.cc:769] Sending connection context
0619 04:33:39.948548 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 04:33:39.949100 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":278,"threads_started":1}
I20260619 04:33:39.951839 3670 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44875
I20260619 04:33:39.952140 3670 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:39.952870 3670 heartbeater.cc:511] Master 127.0.97.62:44875 requested a full tablet report, sending...
I20260619 04:33:39.954888 3481 ts_manager.cc:194] Registered new tserver with Master: 87b14bc5a1fd435d838efd07db80d4d1 (127.0.97.1:40187)
I20260619 04:33:39.956264 3481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 04:33:39.969555 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44875
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:40.277556 3676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:40.277845 3676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:40.277961 3676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:40.285547 3676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:40.285661 3676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:40.285804 3676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:33:40.295670 3676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:40.297015 3676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:40.298240 3676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:40.310604 3681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:40.310676 3682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:40.310631 3684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:40.311543 3676 server_base.cc:1034] running on GCE node
I20260619 04:33:40.312127 3676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:40.313042 3676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:40.314239 3676 hybrid_clock.cc:648] HybridClock initialized: now 1781843620314192 us; error 42 us; skew 500 ppm
I20260619 04:33:40.314538 3676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:40 dist-test-slave-n3tq krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:40.317541 3676 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:33:40.318934 3676 webserver.cc:460] Webserver started at http://127.0.97.2:45777/ using document root <none> and password file <none>
I20260619 04:33:40.319602 3676 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:40.319713 3676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:40.320000 3676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:40.322817 3676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "c211dbf2768943fbb68badffc8b75d4d"
format_stamp: "Formatted at 2026-06-19 04:33:40 on dist-test-slave-n3tq"
server_key: "8c53063818fcaa4cca3c01269b64b4ee"
server_key_iv: "91dd3e1afe524fe552956353591aabfd"
server_key_version: "encryptionkey@0"
I20260619 04:33:40.323558 3676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c211dbf2768943fbb68badffc8b75d4d"
format_stamp: "Formatted at 2026-06-19 04:33:40 on dist-test-slave-n3tq"
server_key: "8c53063818fcaa4cca3c01269b64b4ee"
server_key_iv: "91dd3e1afe524fe552956353591aabfd"
server_key_version: "encryptionkey@0"
I20260619 04:33:40.328575 3676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 04:33:40.331844 3691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:40.333206 3676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:33:40.333333 3676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "c211dbf2768943fbb68badffc8b75d4d"
format_stamp: "Formatted at 2026-06-19 04:33:40 on dist-test-slave-n3tq"
server_key: "8c53063818fcaa4cca3c01269b64b4ee"
server_key_iv: "91dd3e1afe524fe552956353591aabfd"
server_key_version: "encryptionkey@0"
I20260619 04:33:40.333498 3676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:40.362684 3676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:40.366943 3676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:40.367202 3676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:40.368100 3676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:40.369505 3676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:40.369580 3676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:40.369692 3676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:40.369742 3676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:40.402099 3676 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41677
I20260619 04:33:40.402175 3803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41677 every 8 connection(s)
I20260619 04:33:40.403288 3676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:33:40.407567 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 3676
I20260619 04:33:40.407749 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:33:40.408272 388 external_mini_cluster.cc:1428] Setting key a6792c1232d68066e0162b0cb14e9ec4
Jun 19 04:33:40 dist-test-slave-n3tq krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:40.423661 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:40.405587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33217 (local address 127.0.97.62:44875)
0619 04:33:40.405777 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:40.405784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:40.406888 (+ 1104us) server_negotiation.cc:407] Connection header received
0619 04:33:40.409102 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:40.409106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:40.409176 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:40.409260 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:40.410924 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:40.411628 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:40.413623 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:40.414004 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:40.414093 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:40.417622 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:40.417629 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:40.417634 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:40.417669 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:40.420685 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:40.421369 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:40.421375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:40.421378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:40.421520 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:40.421913 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:40.421917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:40.421920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:40.422263 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:40.422406 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:40.423076 (+ 670us) server_negotiation.cc:299] Negotiation successful
0619 04:33:40.423314 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 04:33:40.424430 3806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:40.406081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44875 (local address 127.0.97.2:33217)
0619 04:33:40.406750 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:40.406784 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:40.408854 (+ 2070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:40.409461 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:40.409469 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:40.410023 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:40.410764 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:40.410790 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:40.411773 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:40.411779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:40.413510 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:40.413523 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:40.414193 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:40.414199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:40.414448 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:40.415104 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:40.415131 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:40.417424 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:40.420815 (+ 3391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:40.420822 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:40.420839 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:40.421244 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:40.421621 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:40.421624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:40.421626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:40.421811 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:40.422389 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:40.422396 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:40.422709 (+ 313us) client_negotiation.cc:769] Sending connection context
0619 04:33:40.422933 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 04:33:40.423323 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":245,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 04:33:40.425882 3804 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44875
I20260619 04:33:40.426175 3804 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:40.426898 3804 heartbeater.cc:511] Master 127.0.97.62:44875 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 04:33:40.428371 3481 ts_manager.cc:194] Registered new tserver with Master: c211dbf2768943fbb68badffc8b75d4d (127.0.97.2:41677)
I20260619 04:33:40.429142 3481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 04:33:40.447427 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44875
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:40.755255 3809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:40.755517 3809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:40.755643 3809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:40.762760 3809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:40.762872 3809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:40.763006 3809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:33:40.772763 3809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:40.774034 3809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:40.775281 3809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:40.787359 3814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:40.787438 3817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:40.787482 3815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:40.788249 3809 server_base.cc:1034] running on GCE node
I20260619 04:33:40.788841 3809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:40.789691 3809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:40.790879 3809 hybrid_clock.cc:648] HybridClock initialized: now 1781843620790833 us; error 42 us; skew 500 ppm
I20260619 04:33:40.791162 3809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:40 dist-test-slave-n3tq krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:40.794191 3809 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:33:40.795596 3809 webserver.cc:460] Webserver started at http://127.0.97.3:43803/ using document root <none> and password file <none>
I20260619 04:33:40.796262 3809 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:40.796368 3809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:40.796676 3809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:40.799413 3809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "ce96442c754a449daaca0a00a90c9ac0"
format_stamp: "Formatted at 2026-06-19 04:33:40 on dist-test-slave-n3tq"
server_key: "a55910bc1ca2e3e9b65b8464da76d987"
server_key_iv: "49fdeac9953eee574de0dd0315f9366c"
server_key_version: "encryptionkey@0"
I20260619 04:33:40.800123 3809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ce96442c754a449daaca0a00a90c9ac0"
format_stamp: "Formatted at 2026-06-19 04:33:40 on dist-test-slave-n3tq"
server_key: "a55910bc1ca2e3e9b65b8464da76d987"
server_key_iv: "49fdeac9953eee574de0dd0315f9366c"
server_key_version: "encryptionkey@0"
I20260619 04:33:40.804965 3809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 04:33:40.808203 3824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:40.809489 3809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:33:40.809629 3809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "ce96442c754a449daaca0a00a90c9ac0"
format_stamp: "Formatted at 2026-06-19 04:33:40 on dist-test-slave-n3tq"
server_key: "a55910bc1ca2e3e9b65b8464da76d987"
server_key_iv: "49fdeac9953eee574de0dd0315f9366c"
server_key_version: "encryptionkey@0"
I20260619 04:33:40.809793 3809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:40.856878 3809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:40.860735 3809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:40.861001 3809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:40.861963 3809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:40.863279 3809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:40.863368 3809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:40.863490 3809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:40.863543 3809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:40.892573 3809 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44917
I20260619 04:33:40.892627 3936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44917 every 8 connection(s)
I20260619 04:33:40.893801 3809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:33:40.903780 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 3809
I20260619 04:33:40.903965 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:33:40.904359 388 external_mini_cluster.cc:1428] Setting key 8f733a963688c9c39c71ae4ef05cf3ad
Jun 19 04:33:40 dist-test-slave-n3tq krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:40.912987 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:40.896165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37889 (local address 127.0.97.62:44875)
0619 04:33:40.896331 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:40.896335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:40.897739 (+ 1404us) server_negotiation.cc:407] Connection header received
0619 04:33:40.899996 (+ 2257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:40.900002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:40.900087 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:40.900174 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:40.901756 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:40.902537 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:40.904027 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:40.904484 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:40.904597 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:40.908167 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:40.908172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:40.908175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:40.908208 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:40.910505 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:40.911147 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:40.911150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:40.911153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:40.911260 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:40.911576 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:40.911579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:40.911581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:40.911839 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:40.911940 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:40.912574 (+ 634us) server_negotiation.cc:299] Negotiation successful
0619 04:33:40.912736 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 04:33:40.913961 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:40.896744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44875 (local address 127.0.97.3:37889)
0619 04:33:40.897561 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:40.897610 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:40.899719 (+ 2109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:40.900425 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:40.900433 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:40.900984 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:40.901598 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:40.901614 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:40.902672 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:40.902677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:40.903891 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:40.903904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:40.904686 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:40.904694 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:40.904999 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:40.905717 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:40.905742 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:40.907988 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:40.910633 (+ 2645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:40.910639 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:40.910654 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:40.911056 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:40.911332 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:40.911335 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:40.911337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:40.911503 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:40.911923 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:40.911928 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:40.912230 (+ 302us) client_negotiation.cc:769] Sending connection context
0619 04:33:40.912492 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 04:33:40.912887 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":321,"threads_started":1}
I20260619 04:33:40.915242 3937 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44875
I20260619 04:33:40.915506 3937 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:40.916185 3937 heartbeater.cc:511] Master 127.0.97.62:44875 requested a full tablet report, sending...
I20260619 04:33:40.917616 3481 ts_manager.cc:194] Registered new tserver with Master: ce96442c754a449daaca0a00a90c9ac0 (127.0.97.3:44917)
I20260619 04:33:40.918638 3481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37889
I20260619 04:33:40.919571 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 04:33:40 dist-test-slave-n3tq krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843620, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 04:33:40 dist-test-slave-n3tq krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843620, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:40.958989 3670 heartbeater.cc:503] Master 127.0.97.62:44875 was elected leader, sending a full tablet report...
I20260619 04:33:40.962378 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:40.949521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53610 (local address 127.0.97.62:44875)
0619 04:33:40.949688 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:40.949693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:40.950089 (+ 396us) server_negotiation.cc:407] Connection header received
0619 04:33:40.950234 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:40.950238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:40.950298 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:40.950388 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:40.951700 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:40.952415 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:40.953628 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:40.954018 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:40.954107 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:40.957613 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:40.957618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:40.957621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:40.957653 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:40.960157 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:40.960861 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:40.960864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:40.960867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:40.960991 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:40.961278 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:40.961280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:40.961282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:40.961557 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:40.961726 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:40.961944 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 04:33:40.962104 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 04:33:40.962815 3481 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53610
I20260619 04:33:40.964615 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 3542
Jun 19 04:33:40 dist-test-slave-n3tq krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843620, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 04:33:40.979269 3948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:40.965583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40898 (local address 127.0.97.1:40187)
0619 04:33:40.966152 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:40.966159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:40.966182 (+ 23us) server_negotiation.cc:407] Connection header received
0619 04:33:40.966240 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:40.966246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:40.966439 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:40.966572 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:40.967550 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:40.968567 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:40.969781 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:40.970346 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:40.970460 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:40.974147 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:40.974163 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:40.974182 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:40.974225 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:40.977091 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:40.977700 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:40.977706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:40.977709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:40.977819 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:40.978086 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:40.978088 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:40.978090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:40.978406 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:40.978625 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:40.978804 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 04:33:40.978994 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":281,"threads_started":1}
W20260619 04:33:40.980479 3665 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40898
W20260619 04:33:40.981576 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 3542: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:33:40.981758 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 3542
I20260619 04:33:40.994131 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 3676
Jun 19 04:33:40 dist-test-slave-n3tq krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843620, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 04:33:41.008991 3949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:40.995000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54804 (local address 127.0.97.2:41677)
0619 04:33:40.995503 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:40.995510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:40.995536 (+ 26us) server_negotiation.cc:407] Connection header received
0619 04:33:40.995592 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:40.995598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:40.995830 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:40.995953 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:40.997185 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:40.998184 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:40.999405 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:40.999937 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:41.000029 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:41.003870 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:41.003880 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:41.003894 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:41.003926 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:41.006628 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:41.007268 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:41.007275 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:41.007278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:41.007390 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:41.007649 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:41.007652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:41.007654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:41.007970 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:41.008173 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:41.008407 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 04:33:41.008626 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":258,"threads_started":1}
W20260619 04:33:41.010156 3799 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54804
W20260619 04:33:41.011185 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 3676: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:33:41.011333 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 3676
I20260619 04:33:41.023190 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 3809
Jun 19 04:33:41 dist-test-slave-n3tq krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843620, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 04:33:41.038394 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:41.024089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59266 (local address 127.0.97.3:44917)
0619 04:33:41.024736 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:41.024743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:41.024772 (+ 29us) server_negotiation.cc:407] Connection header received
0619 04:33:41.024828 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:41.024836 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:41.025043 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:41.025191 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:41.026430 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:41.027437 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:41.028710 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:41.029236 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:41.029346 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:41.033315 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:41.033326 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:41.033341 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:41.033372 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:41.035911 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:41.036601 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:41.036608 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:41.036610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:41.036735 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:41.037103 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:41.037107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:41.037109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:41.037456 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:41.037684 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:41.037919 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 04:33:41.038108 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":292,"threads_started":1}
W20260619 04:33:41.039562 3932 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59266
W20260619 04:33:41.040630 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 3809: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:33:41.040766 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 3809
I20260619 04:33:41.052515 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 3449
I20260619 04:33:41.053622 3511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:41.151815 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 3449
2026-06-19T04:33:41Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3355 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:33:41 dist-test-slave-n3tq krb5kdc[3954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:33:41 dist-test-slave-n3tq krb5kdc[3954](info): set up 2 sockets
Jun 19 04:33:41 dist-test-slave-n3tq krb5kdc[3954](info): commencing operation
krb5kdc: starting...
W20260619 04:33:42.132503 388 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:33:42 dist-test-slave-n3tq krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:33:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:33:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:33:42.297097 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33591
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:42.592828 3967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:42.593098 3967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:42.593181 3967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:42.600042 3967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:33:42.600138 3967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:42.600184 3967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:42.600239 3967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:33:42.600279 3967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:33:42.611550 3967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:42.612780 3967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:42.613970 3967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:42.624439 3973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:42.624492 3975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:42.624405 3972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:42.625442 3967 server_base.cc:1034] running on GCE node
I20260619 04:33:42.626142 3967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:42.627408 3967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:42.628587 3967 hybrid_clock.cc:648] HybridClock initialized: now 1781843622628541 us; error 30 us; skew 500 ppm
I20260619 04:33:42.628907 3967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:42 dist-test-slave-n3tq krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:42.637476 3967 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:33:42.638923 3967 webserver.cc:460] Webserver started at http://127.0.97.62:39473/ using document root <none> and password file <none>
I20260619 04:33:42.639593 3967 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:42.639705 3967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:42.640010 3967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:42.642941 3967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "675f11d89e5e402d91ee3d2757ad8532"
format_stamp: "Formatted at 2026-06-19 04:33:42 on dist-test-slave-n3tq"
server_key: "356a7c2495c8c7fa76893ceec9be9854"
server_key_iv: "33e5679bc9818ceaf5fb6a07d4690ac7"
server_key_version: "encryptionkey@0"
I20260619 04:33:42.643680 3967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "675f11d89e5e402d91ee3d2757ad8532"
format_stamp: "Formatted at 2026-06-19 04:33:42 on dist-test-slave-n3tq"
server_key: "356a7c2495c8c7fa76893ceec9be9854"
server_key_iv: "33e5679bc9818ceaf5fb6a07d4690ac7"
server_key_version: "encryptionkey@0"
I20260619 04:33:42.648711 3967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 04:33:42.652024 3982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:42.653414 3967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:33:42.653577 3967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "675f11d89e5e402d91ee3d2757ad8532"
format_stamp: "Formatted at 2026-06-19 04:33:42 on dist-test-slave-n3tq"
server_key: "356a7c2495c8c7fa76893ceec9be9854"
server_key_iv: "33e5679bc9818ceaf5fb6a07d4690ac7"
server_key_version: "encryptionkey@0"
I20260619 04:33:42.653755 3967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:42.669286 3967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:42.672958 3967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:42.673192 3967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:42.689008 3967 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33591
I20260619 04:33:42.689046 4033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33591 every 8 connection(s)
I20260619 04:33:42.690199 3967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:33:42.691998 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 3967
I20260619 04:33:42.692173 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:33:42.692795 388 external_mini_cluster.cc:1428] Setting key 1f40560ebfe2edd05ca316c4e394b27e
I20260619 04:33:42.693734 4034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 04:33:42 dist-test-slave-n3tq krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:42.706436 4034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532: Bootstrap starting.
I20260619 04:33:42.710014 4034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:42.711145 4034 log.cc:824] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:42.714098 4034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532: No bootstrap required, opened a new log
I20260619 04:33:42.714351 4037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:42.694530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41956 (local address 127.0.97.62:33591)
0619 04:33:42.695266 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:42.695280 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:42.695322 (+ 42us) server_negotiation.cc:407] Connection header received
0619 04:33:42.697077 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:42.697103 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:42.697585 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:42.698006 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:42.699173 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:42.701116 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:42.702248 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:42.703189 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:42.703355 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:42.706952 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:42.706961 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:42.706977 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:42.707015 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:42.710496 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:42.711053 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:42.711061 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:42.711069 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:42.711266 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:42.711638 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:42.711644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:42.711648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:42.712170 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:42.712426 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:42.712786 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 04:33:42.713156 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":305,"threads_started":1}
I20260619 04:33:42.721534 4034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675f11d89e5e402d91ee3d2757ad8532" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33591 } }
I20260619 04:33:42.721850 4034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:42.721927 4034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 675f11d89e5e402d91ee3d2757ad8532, State: Initialized, Role: FOLLOWER
I20260619 04:33:42.722373 4034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675f11d89e5e402d91ee3d2757ad8532" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33591 } }
I20260619 04:33:42.722572 4034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:33:42.722719 4034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:33:42.722906 4034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:42.725033 4034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675f11d89e5e402d91ee3d2757ad8532" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33591 } }
I20260619 04:33:42.725473 4034 leader_election.cc:302] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 675f11d89e5e402d91ee3d2757ad8532; no voters:
I20260619 04:33:42.725994 4034 leader_election.cc:288] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:33:42.726105 4039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:42.726717 4039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532 [term 1 LEADER]: Becoming Leader. State: Replica: 675f11d89e5e402d91ee3d2757ad8532, State: Running, Role: LEADER
I20260619 04:33:42.727195 4039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675f11d89e5e402d91ee3d2757ad8532" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33591 } }
I20260619 04:33:42.727809 4034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:33:42.729815 4041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 675f11d89e5e402d91ee3d2757ad8532. Latest consensus state: current_term: 1 leader_uuid: "675f11d89e5e402d91ee3d2757ad8532" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675f11d89e5e402d91ee3d2757ad8532" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33591 } } }
I20260619 04:33:42.730170 4041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532 [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:42.731683 4044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:33:42.731621 4040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "675f11d89e5e402d91ee3d2757ad8532" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675f11d89e5e402d91ee3d2757ad8532" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33591 } } }
I20260619 04:33:42.731904 4040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532 [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:42.736339 4044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:33:42.744632 4044 catalog_manager.cc:1347] Generated new cluster ID: beabd532d1d44e9e9122ce4d8266a310
I20260619 04:33:42.744784 4044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:33:42.765524 4044 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:33:42.766762 4044 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:33:42.784624 4044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 675f11d89e5e402d91ee3d2757ad8532: Generated new TSK 0
I20260619 04:33:42.785467 4044 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 04:33:42.839352 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33591
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:43.146701 4060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:43.146979 4060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:43.147100 4060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:43.154136 4060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:43.154253 4060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:43.154399 4060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:33:43.163909 4060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:43.165175 4060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:43.166419 4060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:43.178318 4066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:43.178355 4065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:43.178375 4068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:43.179042 4060 server_base.cc:1034] running on GCE node
I20260619 04:33:43.179564 4060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:43.180401 4060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:43.181604 4060 hybrid_clock.cc:648] HybridClock initialized: now 1781843623181553 us; error 49 us; skew 500 ppm
I20260619 04:33:43.181874 4060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:43 dist-test-slave-n3tq krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:43.184674 4060 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:33:43.185887 4060 webserver.cc:460] Webserver started at http://127.0.97.1:43649/ using document root <none> and password file <none>
I20260619 04:33:43.186463 4060 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:43.186553 4060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:43.186812 4060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:43.189527 4060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "679091abbdb14bfcaef5f83ed93ff2d4"
format_stamp: "Formatted at 2026-06-19 04:33:43 on dist-test-slave-n3tq"
server_key: "caff4764b5b4c0cff32e8728966a2ec8"
server_key_iv: "e794c0210756a660f33153a8dec41124"
server_key_version: "encryptionkey@0"
I20260619 04:33:43.190239 4060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "679091abbdb14bfcaef5f83ed93ff2d4"
format_stamp: "Formatted at 2026-06-19 04:33:43 on dist-test-slave-n3tq"
server_key: "caff4764b5b4c0cff32e8728966a2ec8"
server_key_iv: "e794c0210756a660f33153a8dec41124"
server_key_version: "encryptionkey@0"
I20260619 04:33:43.194818 4060 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 04:33:43.197887 4075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:43.199043 4060 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 04:33:43.199169 4060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "679091abbdb14bfcaef5f83ed93ff2d4"
format_stamp: "Formatted at 2026-06-19 04:33:43 on dist-test-slave-n3tq"
server_key: "caff4764b5b4c0cff32e8728966a2ec8"
server_key_iv: "e794c0210756a660f33153a8dec41124"
server_key_version: "encryptionkey@0"
I20260619 04:33:43.199342 4060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:43.225265 4060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:43.229198 4060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:43.229442 4060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:43.230309 4060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:43.231590 4060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:43.231660 4060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:43.231772 4060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:43.231823 4060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:43.260769 4060 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32877
I20260619 04:33:43.260808 4187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32877 every 8 connection(s)
I20260619 04:33:43.261930 4060 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:33:43.267088 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 4060
I20260619 04:33:43.267272 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:33:43.268016 388 external_mini_cluster.cc:1428] Setting key e0d56d4e9f9eeae5d904ad02bc4004e2
Jun 19 04:33:43 dist-test-slave-n3tq krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:43.282444 4190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:43.264171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39431 (local address 127.0.97.62:33591)
0619 04:33:43.264618 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:43.264624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:43.265570 (+ 946us) server_negotiation.cc:407] Connection header received
0619 04:33:43.267498 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:43.267502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:43.267570 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:43.267665 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:43.269441 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:43.270232 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:43.272418 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:43.272866 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:43.272978 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:43.276734 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:43.276741 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:43.276745 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:43.276779 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:43.279684 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:43.280332 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:43.280337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:43.280340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:43.280490 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:43.280836 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:43.280841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:43.280844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:43.281171 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:43.281297 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:43.281888 (+ 591us) server_negotiation.cc:299] Negotiation successful
0619 04:33:43.282103 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":206,"threads_started":1}
I20260619 04:33:43.283231 4191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:43.264756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33591 (local address 127.0.97.1:39431)
0619 04:33:43.265416 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:43.265455 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:43.267214 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:43.267916 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:43.267928 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:43.268627 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:43.269241 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:43.269264 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:43.270384 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:43.270390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:43.272206 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:43.272220 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:43.273083 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:43.273089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:43.273338 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:43.274023 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:43.274049 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:43.276530 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:43.279816 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:43.279822 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:43.279835 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:43.280223 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:43.280572 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:43.280575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:43.280577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:43.280739 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:43.281270 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:43.281276 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:43.281568 (+ 292us) client_negotiation.cc:769] Sending connection context
0619 04:33:43.281772 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 04:33:43.282139 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":264,"threads_started":1}
I20260619 04:33:43.284734 4188 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33591
I20260619 04:33:43.285025 4188 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:43.285678 4188 heartbeater.cc:511] Master 127.0.97.62:33591 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 04:33:43.287693 3999 ts_manager.cc:194] Registered new tserver with Master: 679091abbdb14bfcaef5f83ed93ff2d4 (127.0.97.1:32877)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 04:33:43.289171 3999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 04:33:43.307443 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33591
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:43.606163 4194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:43.606420 4194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:43.606518 4194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:43.613421 4194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:43.613516 4194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:43.613631 4194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:33:43.622892 4194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:43.623984 4194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:43.625187 4194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:43.636811 4200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:43.636878 4199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:43.636857 4202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:43.637677 4194 server_base.cc:1034] running on GCE node
I20260619 04:33:43.638232 4194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:43.639084 4194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:43.640257 4194 hybrid_clock.cc:648] HybridClock initialized: now 1781843623640219 us; error 31 us; skew 500 ppm
I20260619 04:33:43.640561 4194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:43 dist-test-slave-n3tq krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:43.643268 4194 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:33:43.644590 4194 webserver.cc:460] Webserver started at http://127.0.97.2:42313/ using document root <none> and password file <none>
I20260619 04:33:43.645208 4194 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:43.645306 4194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:43.645577 4194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:43.648164 4194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "8da9d5f626a746d08dd58f3bdef8825a"
format_stamp: "Formatted at 2026-06-19 04:33:43 on dist-test-slave-n3tq"
server_key: "1ed942cea3b6946364600edb28eb0c95"
server_key_iv: "1ceb1b4fa13d45f46276466b6fd59953"
server_key_version: "encryptionkey@0"
I20260619 04:33:43.648907 4194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8da9d5f626a746d08dd58f3bdef8825a"
format_stamp: "Formatted at 2026-06-19 04:33:43 on dist-test-slave-n3tq"
server_key: "1ed942cea3b6946364600edb28eb0c95"
server_key_iv: "1ceb1b4fa13d45f46276466b6fd59953"
server_key_version: "encryptionkey@0"
I20260619 04:33:43.653555 4194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 04:33:43.656698 4209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:43.657948 4194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:33:43.658083 4194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "8da9d5f626a746d08dd58f3bdef8825a"
format_stamp: "Formatted at 2026-06-19 04:33:43 on dist-test-slave-n3tq"
server_key: "1ed942cea3b6946364600edb28eb0c95"
server_key_iv: "1ceb1b4fa13d45f46276466b6fd59953"
server_key_version: "encryptionkey@0"
I20260619 04:33:43.658257 4194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:43.676018 4194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:43.679611 4194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:43.679850 4194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:43.680757 4194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:43.681996 4194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:43.682081 4194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:43.682199 4194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:43.682245 4194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:43.713191 4194 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45627
I20260619 04:33:43.713232 4321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45627 every 8 connection(s)
I20260619 04:33:43.714393 4194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:33:43.722983 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 4194
I20260619 04:33:43.723153 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:33:43.723567 388 external_mini_cluster.cc:1428] Setting key 34f368e4899cbe494e4a24f102c126bf
Jun 19 04:33:43 dist-test-slave-n3tq krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:43.734210 4190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:43.716765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52229 (local address 127.0.97.62:33591)
0619 04:33:43.716922 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:43.716926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:43.718107 (+ 1181us) server_negotiation.cc:407] Connection header received
0619 04:33:43.719785 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:43.719790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:43.719863 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:43.719957 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:43.721444 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:43.722199 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:43.723897 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:43.724405 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:43.724568 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:43.728267 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:43.728274 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:43.728278 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:43.728320 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:43.731219 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:43.732136 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:43.732140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:43.732142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:43.732255 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:43.732695 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:43.732698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:43.732700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:43.732942 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:43.733053 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:43.733807 (+ 754us) server_negotiation.cc:299] Negotiation successful
0619 04:33:43.733968 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 04:33:43.735607 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:43.717259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33591 (local address 127.0.97.2:52229)
0619 04:33:43.717930 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:43.717978 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:43.719520 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:43.720155 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:43.720163 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:43.720702 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:43.721265 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:43.721283 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:43.722400 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:43.722405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:43.723752 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:43.723765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:43.724669 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:43.724677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:43.724975 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:43.725704 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:43.725731 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:43.728054 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:43.731371 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:43.731382 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:43.731403 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:43.732030 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:43.732354 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:43.732359 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:43.732362 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:43.732600 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:43.733053 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:43.733060 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:43.733471 (+ 411us) client_negotiation.cc:769] Sending connection context
0619 04:33:43.733759 (+ 288us) client_negotiation.cc:240] Negotiation successful
0619 04:33:43.734291 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":264,"threads_started":1}
I20260619 04:33:43.737010 4322 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33591
I20260619 04:33:43.737318 4322 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:43.738206 4322 heartbeater.cc:511] Master 127.0.97.62:33591 requested a full tablet report, sending...
I20260619 04:33:43.739759 3999 ts_manager.cc:194] Registered new tserver with Master: 8da9d5f626a746d08dd58f3bdef8825a (127.0.97.2:45627)
I20260619 04:33:43.740592 3999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52229
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 04:33:43.763831 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33591
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:44.066984 4327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:44.067250 4327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:44.067344 4327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:44.074349 4327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:44.074452 4327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:44.074568 4327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:33:44.083938 4327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:44.085124 4327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:44.086329 4327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:44.098155 4332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:44.098130 4333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:44.098330 4335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:44.098927 4327 server_base.cc:1034] running on GCE node
I20260619 04:33:44.099472 4327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:44.100337 4327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:44.101528 4327 hybrid_clock.cc:648] HybridClock initialized: now 1781843624101488 us; error 56 us; skew 500 ppm
I20260619 04:33:44.101812 4327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:44 dist-test-slave-n3tq krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:44.104674 4327 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:33:44.105947 4327 webserver.cc:460] Webserver started at http://127.0.97.3:39625/ using document root <none> and password file <none>
I20260619 04:33:44.106570 4327 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:44.106679 4327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:44.106945 4327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:44.109723 4327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "d514d4f01e8a4f8e9ae5660a6d51e135"
format_stamp: "Formatted at 2026-06-19 04:33:44 on dist-test-slave-n3tq"
server_key: "0e10dd05eca7e0462b81ba615ff813c1"
server_key_iv: "17b0a9d434c12294b4efcac652aa776b"
server_key_version: "encryptionkey@0"
I20260619 04:33:44.110427 4327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d514d4f01e8a4f8e9ae5660a6d51e135"
format_stamp: "Formatted at 2026-06-19 04:33:44 on dist-test-slave-n3tq"
server_key: "0e10dd05eca7e0462b81ba615ff813c1"
server_key_iv: "17b0a9d434c12294b4efcac652aa776b"
server_key_version: "encryptionkey@0"
I20260619 04:33:44.115099 4327 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 04:33:44.118319 4342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:44.119598 4327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:33:44.119730 4327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "d514d4f01e8a4f8e9ae5660a6d51e135"
format_stamp: "Formatted at 2026-06-19 04:33:44 on dist-test-slave-n3tq"
server_key: "0e10dd05eca7e0462b81ba615ff813c1"
server_key_iv: "17b0a9d434c12294b4efcac652aa776b"
server_key_version: "encryptionkey@0"
I20260619 04:33:44.119904 4327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:44.137542 4327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:44.141405 4327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:44.141651 4327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:44.142536 4327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:44.143878 4327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:44.144011 4327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:44.144164 4327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:44.144227 4327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:44.171927 4327 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36711
I20260619 04:33:44.171963 4454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36711 every 8 connection(s)
I20260619 04:33:44.173123 4327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:33:44.179407 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 4327
I20260619 04:33:44.179569 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:33:44.180090 388 external_mini_cluster.cc:1428] Setting key 243af72fc68dca6c01ab904b75d239eb
Jun 19 04:33:44 dist-test-slave-n3tq krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:44.191757 4190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:44.175408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52883 (local address 127.0.97.62:33591)
0619 04:33:44.175572 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:44.175577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:44.176760 (+ 1183us) server_negotiation.cc:407] Connection header received
0619 04:33:44.178469 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:44.178474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:44.178553 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:44.178639 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:44.180431 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:44.181276 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:44.183112 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:44.183511 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:44.183604 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:44.186742 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:44.186747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:44.186750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:44.186780 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:44.189193 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:44.189842 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:44.189846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:44.189849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:44.189982 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:44.190397 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:44.190401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:44.190403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:44.190667 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:44.190774 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:44.191373 (+ 599us) server_negotiation.cc:299] Negotiation successful
0619 04:33:44.191520 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 04:33:44.192713 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:44.175912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33591 (local address 127.0.97.3:52883)
0619 04:33:44.176612 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:44.176654 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:44.178219 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:44.178866 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:44.178874 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:44.179528 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:44.180264 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:44.180290 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:44.181408 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:44.181411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:44.182980 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:44.182991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:44.183678 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:44.183682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:44.183860 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:44.184576 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:44.184601 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:44.186548 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:44.189314 (+ 2766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:44.189320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:44.189332 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:44.189730 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:44.190098 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:44.190101 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:44.190104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:44.190272 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:44.190771 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:44.190776 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:44.191072 (+ 296us) client_negotiation.cc:769] Sending connection context
0619 04:33:44.191285 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 04:33:44.191632 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":258,"threads_started":1}
I20260619 04:33:44.194096 4455 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33591
I20260619 04:33:44.194373 4455 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:44.195117 4455 heartbeater.cc:511] Master 127.0.97.62:33591 requested a full tablet report, sending...
I20260619 04:33:44.196429 3998 ts_manager.cc:194] Registered new tserver with Master: d514d4f01e8a4f8e9ae5660a6d51e135 (127.0.97.3:36711)
I20260619 04:33:44.197250 3998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52883
I20260619 04:33:44.206692 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 04:33:44 dist-test-slave-n3tq krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 04:33:44 dist-test-slave-n3tq krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 04:33:44.248663 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:44.234520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41586 (local address 127.0.97.1:32877)
0619 04:33:44.235177 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:44.235185 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:44.235211 (+ 26us) server_negotiation.cc:407] Connection header received
0619 04:33:44.235270 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:44.235276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:44.235496 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:44.235621 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:44.237083 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:44.238112 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:44.239290 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:44.239832 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:44.239934 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:44.243578 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:44.243590 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:44.243604 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:44.243638 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:44.246128 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:44.246705 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:44.246712 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:44.246714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:44.246827 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:44.247168 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:44.247171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:44.247173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:44.247503 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:44.247800 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:44.248061 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 04:33:44.248297 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":409,"threads_started":1}
Jun 19 04:33:44 dist-test-slave-n3tq krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:44.265507 4190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:44.252778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41966 (local address 127.0.97.62:33591)
0619 04:33:44.252976 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:44.252982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:44.253124 (+ 142us) server_negotiation.cc:407] Connection header received
0619 04:33:44.253277 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:44.253281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:44.253340 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:44.253417 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:44.254546 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:44.255229 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:44.256380 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:44.256852 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:44.256956 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:44.260558 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:44.260566 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:44.260571 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:44.260610 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:44.263170 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:44.263871 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:44.263877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:44.263879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:44.263998 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:44.264305 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:44.264308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:44.264310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:44.264641 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:44.264810 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:44.265114 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 04:33:44.265261 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 04:33:44.266003 3998 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41966
I20260619 04:33:44.266964 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 4060
I20260619 04:33:44.279796 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:44.267727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41598 (local address 127.0.97.1:32877)
0619 04:33:44.267886 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:44.267893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:44.269363 (+ 1470us) server_negotiation.cc:407] Connection header received
0619 04:33:44.269505 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:44.269510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:44.269592 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:44.269670 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:44.270793 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:44.271443 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:44.272760 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:44.273330 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:44.273441 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:44.275465 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:44.275472 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:44.275478 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:44.275516 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:44.277730 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:44.278291 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:44.278297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:44.278300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:44.278414 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:44.278719 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:44.278723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:44.278725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:44.278980 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:44.279151 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:44.279342 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 04:33:44.279506 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 04:33:44.280371 4183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:44.358989 4188 heartbeater.cc:503] Master 127.0.97.62:33591 was elected leader, sending a full tablet report...
I20260619 04:33:44.359615 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 4060
W20260619 04:33:44.372339 3986 connection.cc:535] server connection from 127.0.97.1:39431 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 04:33:44.372668 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 4194
Jun 19 04:33:44 dist-test-slave-n3tq krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 04:33:44.386945 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:44.373489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55888 (local address 127.0.97.2:45627)
0619 04:33:44.373973 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:44.373980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:44.374004 (+ 24us) server_negotiation.cc:407] Connection header received
0619 04:33:44.374060 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:44.374065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:44.374326 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:44.374456 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:44.375549 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:44.376672 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:44.377972 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:44.378496 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:44.378590 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:44.382255 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:44.382265 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:44.382279 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:44.382309 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:44.384821 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:44.385372 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:44.385378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:44.385381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:44.385488 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:44.385745 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:44.385748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:44.385750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:44.386062 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:44.386275 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:44.386478 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 04:33:44.386664 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":222,"threads_started":1}
I20260619 04:33:44.388072 4317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:44.468222 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 4194
I20260619 04:33:44.480322 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 4327
Jun 19 04:33:44 dist-test-slave-n3tq krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 04:33:44.495328 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:44.481274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60156 (local address 127.0.97.3:36711)
0619 04:33:44.481869 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:44.481876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:44.481903 (+ 27us) server_negotiation.cc:407] Connection header received
0619 04:33:44.481962 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:44.481968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:44.482176 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:44.482328 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:44.483498 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:44.484515 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:44.485725 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:44.486309 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:44.486419 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:44.490364 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:44.490374 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:44.490389 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:44.490423 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:44.492825 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:44.493428 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:44.493435 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:44.493437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:44.493568 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:44.493879 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:44.493883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:44.493885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:44.494220 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:44.494495 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:44.494780 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 04:33:44.495034 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":309,"threads_started":1}
I20260619 04:33:44.496603 4450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:44.576093 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 4327
I20260619 04:33:44.588172 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 3967
I20260619 04:33:44.589248 4029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:44.692303 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 3967
2026-06-19T04:33:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3540 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:33:44 dist-test-slave-n3tq krb5kdc[4474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:33:44 dist-test-slave-n3tq krb5kdc[4474](info): set up 2 sockets
Jun 19 04:33:44 dist-test-slave-n3tq krb5kdc[4474](info): commencing operation
krb5kdc: starting...
W20260619 04:33:45.696445 388 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:33:45 dist-test-slave-n3tq krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:33:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:33:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:33:45.867213 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42619
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:46.167299 4487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:46.167599 4487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:46.167722 4487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:46.174675 4487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:33:46.174793 4487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:46.174863 4487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:46.174935 4487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:33:46.174996 4487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:33:46.184599 4487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:46.185809 4487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:46.187085 4487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:46.197760 4492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:46.197767 4493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:46.198182 4495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:46.198661 4487 server_base.cc:1034] running on GCE node
I20260619 04:33:46.199323 4487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:46.200608 4487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:46.201797 4487 hybrid_clock.cc:648] HybridClock initialized: now 1781843626201755 us; error 37 us; skew 500 ppm
I20260619 04:33:46.202108 4487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:46 dist-test-slave-n3tq krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:46.204856 4487 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:33:46.205865 4487 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:46.205960 4487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:46.206257 4487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:46.209071 4487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "fd163f9fed274d319d90f8ce3edbdbc3"
format_stamp: "Formatted at 2026-06-19 04:33:46 on dist-test-slave-n3tq"
server_key: "49ade3a400f72b2b51fbe8a1b8192685"
server_key_iv: "54952831dc5e0af62852fcba4a91549a"
server_key_version: "encryptionkey@0"
I20260619 04:33:46.209795 4487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "fd163f9fed274d319d90f8ce3edbdbc3"
format_stamp: "Formatted at 2026-06-19 04:33:46 on dist-test-slave-n3tq"
server_key: "49ade3a400f72b2b51fbe8a1b8192685"
server_key_iv: "54952831dc5e0af62852fcba4a91549a"
server_key_version: "encryptionkey@0"
I20260619 04:33:46.214540 4487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260619 04:33:46.217849 4501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:46.219046 4487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:33:46.219204 4487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "fd163f9fed274d319d90f8ce3edbdbc3"
format_stamp: "Formatted at 2026-06-19 04:33:46 on dist-test-slave-n3tq"
server_key: "49ade3a400f72b2b51fbe8a1b8192685"
server_key_iv: "54952831dc5e0af62852fcba4a91549a"
server_key_version: "encryptionkey@0"
I20260619 04:33:46.219394 4487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:46.234351 4487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:46.238252 4487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:46.238499 4487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:46.254758 4487 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42619
I20260619 04:33:46.254777 4552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42619 every 8 connection(s)
I20260619 04:33:46.255584 4487 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:33:46.259043 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:46.263145 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 4487
I20260619 04:33:46.263331 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:33:46.264072 388 external_mini_cluster.cc:1428] Setting key 6387c98e2add01017bd1c28b92330caf
I20260619 04:33:46.269861 4553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3: Bootstrap starting.
I20260619 04:33:46.272769 4553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:46.273710 4553 log.cc:824] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:46.276345 4553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3: No bootstrap required, opened a new log
Jun 19 04:33:46 dist-test-slave-n3tq krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:46.286446 4556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:46.266119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58320 (local address 127.0.97.62:42619)
0619 04:33:46.266889 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:46.266904 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:46.266946 (+ 42us) server_negotiation.cc:407] Connection header received
0619 04:33:46.268688 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:46.268709 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:46.269077 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:46.269430 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:46.270846 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:46.272176 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:46.273517 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:46.274259 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:46.274483 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:46.278179 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:46.278189 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:46.278213 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:46.278264 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:46.282074 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:46.282726 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:46.282734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:46.282743 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:46.282965 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:46.283306 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:46.283311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:46.283315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:46.284034 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:46.284339 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:46.284671 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 04:33:46.285130 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":358,"threads_started":1}
I20260619 04:33:46.286203 4553 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd163f9fed274d319d90f8ce3edbdbc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42619 } }
I20260619 04:33:46.286628 4553 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:46.286760 4553 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd163f9fed274d319d90f8ce3edbdbc3, State: Initialized, Role: FOLLOWER
I20260619 04:33:46.287321 4553 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd163f9fed274d319d90f8ce3edbdbc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42619 } }
I20260619 04:33:46.287524 4553 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:33:46.287650 4553 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:33:46.287828 4553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:46.290541 4553 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd163f9fed274d319d90f8ce3edbdbc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42619 } }
I20260619 04:33:46.290956 4553 leader_election.cc:302] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd163f9fed274d319d90f8ce3edbdbc3; no voters:
I20260619 04:33:46.291452 4553 leader_election.cc:288] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:33:46.291570 4558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:46.292150 4558 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3 [term 1 LEADER]: Becoming Leader. State: Replica: fd163f9fed274d319d90f8ce3edbdbc3, State: Running, Role: LEADER
I20260619 04:33:46.292678 4558 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd163f9fed274d319d90f8ce3edbdbc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42619 } }
I20260619 04:33:46.293331 4553 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:33:46.295369 4560 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd163f9fed274d319d90f8ce3edbdbc3. Latest consensus state: current_term: 1 leader_uuid: "fd163f9fed274d319d90f8ce3edbdbc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd163f9fed274d319d90f8ce3edbdbc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42619 } } }
I20260619 04:33:46.295727 4560 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3 [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:46.300873 4559 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd163f9fed274d319d90f8ce3edbdbc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd163f9fed274d319d90f8ce3edbdbc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42619 } } }
I20260619 04:33:46.301415 4559 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3 [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:46.302491 4567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:33:46.307436 4567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:33:46.314893 4567 catalog_manager.cc:1347] Generated new cluster ID: fd86434ab8f3409f94b6046cf604cd7b
I20260619 04:33:46.315011 4567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:33:46.329041 4567 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:33:46.330246 4567 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:33:46.340698 4567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3: Generated new TSK 0
I20260619 04:33:46.341687 4567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 04:33:46.383731 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42619
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:46.689723 4579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:46.690006 4579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:46.690153 4579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:46.697393 4579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:46.697515 4579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:46.697669 4579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:33:46.707419 4579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:46.708782 4579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:46.710062 4579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:46.721838 4584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:46.721920 4585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:46.722004 4587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:46.722677 4579 server_base.cc:1034] running on GCE node
I20260619 04:33:46.723238 4579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:46.724098 4579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:46.725324 4579 hybrid_clock.cc:648] HybridClock initialized: now 1781843626725275 us; error 36 us; skew 500 ppm
I20260619 04:33:46.725623 4579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:46 dist-test-slave-n3tq krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:46.728384 4579 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:33:46.729431 4579 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:46.729535 4579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:46.729821 4579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:46.732575 4579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "2538c7734fc44e6ea2c8daed2e49972a"
format_stamp: "Formatted at 2026-06-19 04:33:46 on dist-test-slave-n3tq"
server_key: "b3c2de4740f54751e30ba54271e7d357"
server_key_iv: "d00c304f639d07681c72173376bcbcf8"
server_key_version: "encryptionkey@0"
I20260619 04:33:46.733327 4579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2538c7734fc44e6ea2c8daed2e49972a"
format_stamp: "Formatted at 2026-06-19 04:33:46 on dist-test-slave-n3tq"
server_key: "b3c2de4740f54751e30ba54271e7d357"
server_key_iv: "d00c304f639d07681c72173376bcbcf8"
server_key_version: "encryptionkey@0"
I20260619 04:33:46.737951 4579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 04:33:46.741178 4593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:46.742378 4579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:33:46.742513 4579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "2538c7734fc44e6ea2c8daed2e49972a"
format_stamp: "Formatted at 2026-06-19 04:33:46 on dist-test-slave-n3tq"
server_key: "b3c2de4740f54751e30ba54271e7d357"
server_key_iv: "d00c304f639d07681c72173376bcbcf8"
server_key_version: "encryptionkey@0"
I20260619 04:33:46.742705 4579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:46.757436 4579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:46.761006 4579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:46.761240 4579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:46.762115 4579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:46.763360 4579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:46.763448 4579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:46.763556 4579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:46.763626 4579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:46.791865 4579 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40585
I20260619 04:33:46.791903 4705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40585 every 8 connection(s)
I20260619 04:33:46.792673 4579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:33:46.802095 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 4579
I20260619 04:33:46.802260 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:33:46.802956 388 external_mini_cluster.cc:1428] Setting key 99e8f46d6adf6d7bc9218f685bcdf97d
Jun 19 04:33:46 dist-test-slave-n3tq krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:46.811002 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:46.795007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49649 (local address 127.0.97.62:42619)
0619 04:33:46.795401 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:46.795405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:46.796361 (+ 956us) server_negotiation.cc:407] Connection header received
0619 04:33:46.798089 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:46.798095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:46.798169 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:46.798259 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:46.799625 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:46.800370 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:46.801973 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:46.802480 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:46.802579 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:46.805957 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:46.805961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:46.805965 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:46.805994 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:46.808219 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:46.808921 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:46.808925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:46.808927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:46.809032 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:46.809413 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:46.809416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:46.809418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:46.809671 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:46.809793 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:46.810455 (+ 662us) server_negotiation.cc:299] Negotiation successful
0619 04:33:46.810666 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":185,"threads_started":1}
I20260619 04:33:46.811933 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:46.795562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42619 (local address 127.0.97.1:49649)
0619 04:33:46.796219 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:46.796259 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:46.797842 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:46.798482 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:46.798490 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:46.798949 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:46.799470 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:46.799488 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:46.800527 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:46.800531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:46.801836 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:46.801852 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:46.802715 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:46.802723 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:46.803011 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:46.803637 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:46.803663 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:46.805755 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:46.808345 (+ 2590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:46.808353 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:46.808369 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:46.808807 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:46.809136 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:46.809141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:46.809145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:46.809307 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:46.809786 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:46.809793 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:46.810127 (+ 334us) client_negotiation.cc:769] Sending connection context
0619 04:33:46.810348 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 04:33:46.810832 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":259,"threads_started":1}
I20260619 04:33:46.813308 4706 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42619
I20260619 04:33:46.813627 4706 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:46.814328 4706 heartbeater.cc:511] Master 127.0.97.62:42619 requested a full tablet report, sending...
I20260619 04:33:46.816324 4518 ts_manager.cc:194] Registered new tserver with Master: 2538c7734fc44e6ea2c8daed2e49972a (127.0.97.1:40585)
I20260619 04:33:46.817723 4518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49649
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 04:33:46.839143 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42619
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:47.142969 4712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:47.143268 4712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:47.143374 4712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:47.150312 4712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:47.150405 4712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:47.150521 4712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:33:47.159755 4712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:47.160938 4712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:47.162111 4712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:47.173600 4717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:47.173545 4718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:47.173631 4720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:47.174412 4712 server_base.cc:1034] running on GCE node
I20260619 04:33:47.174930 4712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:47.175791 4712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:47.176970 4712 hybrid_clock.cc:648] HybridClock initialized: now 1781843627176925 us; error 46 us; skew 500 ppm
I20260619 04:33:47.177255 4712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:47 dist-test-slave-n3tq krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:47.180142 4712 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:33:47.181363 4712 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:47.181447 4712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:47.181735 4712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:47.184516 4712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "368762b3618e4991b852b858fcb1bc2c"
format_stamp: "Formatted at 2026-06-19 04:33:47 on dist-test-slave-n3tq"
server_key: "280ba504f0557056c38fb986c15013cf"
server_key_iv: "acabf87f962286d0d4a64f41038da3c7"
server_key_version: "encryptionkey@0"
I20260619 04:33:47.185259 4712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "368762b3618e4991b852b858fcb1bc2c"
format_stamp: "Formatted at 2026-06-19 04:33:47 on dist-test-slave-n3tq"
server_key: "280ba504f0557056c38fb986c15013cf"
server_key_iv: "acabf87f962286d0d4a64f41038da3c7"
server_key_version: "encryptionkey@0"
I20260619 04:33:47.189998 4712 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 04:33:47.193364 4726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:47.194686 4712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:33:47.194842 4712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "368762b3618e4991b852b858fcb1bc2c"
format_stamp: "Formatted at 2026-06-19 04:33:47 on dist-test-slave-n3tq"
server_key: "280ba504f0557056c38fb986c15013cf"
server_key_iv: "acabf87f962286d0d4a64f41038da3c7"
server_key_version: "encryptionkey@0"
I20260619 04:33:47.195027 4712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:47.215723 4712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:47.219331 4712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:47.219568 4712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:47.220424 4712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:47.221750 4712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:47.221833 4712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:47.221910 4712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:47.221961 4712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:47.250360 4712 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42011
I20260619 04:33:47.250406 4838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42011 every 8 connection(s)
I20260619 04:33:47.251123 4712 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:33:47.255055 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 4712
I20260619 04:33:47.255219 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:33:47.255594 388 external_mini_cluster.cc:1428] Setting key 02218f2eda7f5a7ce9a593aceb7a39e5
Jun 19 04:33:47 dist-test-slave-n3tq krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:47.272505 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:47.253537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42957 (local address 127.0.97.62:42619)
0619 04:33:47.253704 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:47.253708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:47.255011 (+ 1303us) server_negotiation.cc:407] Connection header received
0619 04:33:47.257535 (+ 2524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:47.257540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:47.257659 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:47.257770 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:47.259582 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.260583 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.262764 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.263371 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.263468 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:47.266835 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:47.266839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:47.266843 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:47.266871 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:47.269580 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:47.270253 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:47.270258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:47.270262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:47.270397 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:47.270781 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:47.270785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:47.270787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:47.271053 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:47.271178 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:47.271880 (+ 702us) server_negotiation.cc:299] Negotiation successful
0619 04:33:47.272156 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 04:33:47.273524 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:47.254079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42619 (local address 127.0.97.2:42957)
0619 04:33:47.254867 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:47.254910 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:47.257244 (+ 2334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:47.257997 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:47.258008 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:47.258663 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:47.259383 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:47.259409 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.260735 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.260740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:47.262617 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:47.262630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.263575 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.263582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:47.263847 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:47.264575 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:47.264598 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:47.266639 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:47.269710 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:47.269719 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:47.269737 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:47.270143 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:47.270502 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:47.270507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:47.270511 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:47.270677 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:47.271178 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:47.271185 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:47.271547 (+ 362us) client_negotiation.cc:769] Sending connection context
0619 04:33:47.271793 (+ 246us) client_negotiation.cc:240] Negotiation successful
0619 04:33:47.272302 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":293,"threads_started":1}
I20260619 04:33:47.274860 4839 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42619
I20260619 04:33:47.275207 4839 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 04:33:47.276003 4839 heartbeater.cc:511] Master 127.0.97.62:42619 requested a full tablet report, sending...
I20260619 04:33:47.277390 4518 ts_manager.cc:194] Registered new tserver with Master: 368762b3618e4991b852b858fcb1bc2c (127.0.97.2:42011)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 04:33:47.278385 4518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 04:33:47.297350 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42619
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:47.597471 4844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:47.597777 4844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:47.597915 4844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:47.605049 4844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:47.605173 4844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:47.605326 4844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:33:47.614640 4844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:47.615877 4844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:47.617152 4844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:47.628782 4849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:47.628816 4852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:47.628829 4850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:47.629861 4844 server_base.cc:1034] running on GCE node
I20260619 04:33:47.630379 4844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:47.631166 4844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:47.632351 4844 hybrid_clock.cc:648] HybridClock initialized: now 1781843627632306 us; error 38 us; skew 500 ppm
I20260619 04:33:47.632660 4844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:47 dist-test-slave-n3tq krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:47.635416 4844 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:33:47.636476 4844 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:47.636577 4844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:47.636857 4844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:47.639483 4844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "326635b3942e44dc9c32a73d520a0424"
format_stamp: "Formatted at 2026-06-19 04:33:47 on dist-test-slave-n3tq"
server_key: "7c91165f20b9ec30131fb576087f14ae"
server_key_iv: "434ab03d86e45a105d731490e472b07e"
server_key_version: "encryptionkey@0"
I20260619 04:33:47.640218 4844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "326635b3942e44dc9c32a73d520a0424"
format_stamp: "Formatted at 2026-06-19 04:33:47 on dist-test-slave-n3tq"
server_key: "7c91165f20b9ec30131fb576087f14ae"
server_key_iv: "434ab03d86e45a105d731490e472b07e"
server_key_version: "encryptionkey@0"
I20260619 04:33:47.644682 4844 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 04:33:47.647854 4858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:47.649068 4844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 04:33:47.649195 4844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "326635b3942e44dc9c32a73d520a0424"
format_stamp: "Formatted at 2026-06-19 04:33:47 on dist-test-slave-n3tq"
server_key: "7c91165f20b9ec30131fb576087f14ae"
server_key_iv: "434ab03d86e45a105d731490e472b07e"
server_key_version: "encryptionkey@0"
I20260619 04:33:47.649374 4844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:47.674350 4844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:47.678148 4844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:47.678377 4844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:47.679242 4844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:47.680482 4844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:47.680578 4844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:47.680683 4844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:47.680760 4844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:47.709582 4844 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43783
I20260619 04:33:47.709631 4970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43783 every 8 connection(s)
I20260619 04:33:47.710356 4844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:33:47.712723 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 4844
I20260619 04:33:47.712904 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:33:47.713339 388 external_mini_cluster.cc:1428] Setting key 56bb3c750a93c61a39359f5c22553e84
Jun 19 04:33:47 dist-test-slave-n3tq krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:47.729457 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:47.712577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37803 (local address 127.0.97.62:42619)
0619 04:33:47.712770 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:47.712776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:47.714031 (+ 1255us) server_negotiation.cc:407] Connection header received
0619 04:33:47.716162 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:47.716168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:47.716245 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:47.716330 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:47.717763 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.718625 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.720109 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.720535 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.720661 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:47.723997 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:47.724003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:47.724006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:47.724040 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:47.726247 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:47.727082 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:47.727086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:47.727088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:47.727218 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:47.727679 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:47.727683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:47.727686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:47.727988 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:47.728123 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:47.728893 (+ 770us) server_negotiation.cc:299] Negotiation successful
0619 04:33:47.729111 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 04:33:47.730415 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:47.713094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42619 (local address 127.0.97.3:37803)
0619 04:33:47.713858 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:47.713913 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:47.715882 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:47.716587 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:47.716595 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:47.717072 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:47.717614 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:47.717631 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.718763 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.718769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:47.719979 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:47.719991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.720748 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.720755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:47.720974 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:47.721585 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:47.721606 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:47.723774 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:47.726395 (+ 2621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:47.726402 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:47.726414 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:47.726946 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:47.727329 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:47.727332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:47.727335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:47.727560 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:47.728123 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:47.728131 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:47.728545 (+ 414us) client_negotiation.cc:769] Sending connection context
0619 04:33:47.728793 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 04:33:47.729314 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":277,"threads_started":1}
I20260619 04:33:47.731595 4971 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42619
I20260619 04:33:47.731865 4971 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:47.732483 4971 heartbeater.cc:511] Master 127.0.97.62:42619 requested a full tablet report, sending...
I20260619 04:33:47.733748 4518 ts_manager.cc:194] Registered new tserver with Master: 326635b3942e44dc9c32a73d520a0424 (127.0.97.3:43783)
I20260619 04:33:47.734491 4518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37803
I20260619 04:33:47.740244 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:33:47.755802 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:47.744343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.0.97.62:42619)
0619 04:33:47.744576 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:47.744582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:47.744992 (+ 410us) server_negotiation.cc:407] Connection header received
0619 04:33:47.745102 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:47.745106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:47.745164 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:47.745241 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:47.746602 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.747428 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.748671 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.749091 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.749181 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:47.751136 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:47.751141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:47.751144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:47.751174 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:47.753614 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:47.754169 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:47.754172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:47.754175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:47.754278 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:47.754567 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:47.754571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:47.754573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:47.754887 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:47.755067 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:47.755337 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 04:33:47.755492 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 04:33:47.759353 4518 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:33:47.765774 4518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:33:47.787048 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:47.778220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.0.97.3:43783)
0619 04:33:47.778841 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:47.778849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:47.778882 (+ 33us) server_negotiation.cc:407] Connection header received
0619 04:33:47.778978 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:47.778985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:47.779245 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:47.779421 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:47.780074 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.782282 (+ 2208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.785000 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.785926 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.786044 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:47.786313 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:47.786548 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 04:33:47.786713 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":398,"threads_started":1}
I20260619 04:33:47.787106 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:47.777270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42011 (local address 127.0.0.1:34434)
0619 04:33:47.777966 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:47.778003 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:47.778167 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:47.779742 (+ 1575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:47.779748 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:47.779778 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:47.780174 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:47.780185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.782478 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.782484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:47.784501 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:47.784512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.786309 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.786316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:47.786470 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:47.786502 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 04:33:47.786561 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 04:33:47.786686 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":52,"thread_start_us":535,"threads_started":1}
I20260619 04:33:47.787528 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:47.777659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34434 (local address 127.0.97.2:42011)
0619 04:33:47.779162 (+ 1503us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:47.779172 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:47.779198 (+ 26us) server_negotiation.cc:407] Connection header received
0619 04:33:47.779292 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:47.779299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:47.779526 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:47.780377 (+ 851us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:47.780624 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.782312 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.784685 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.786070 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.786800 (+ 730us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:47.786981 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:47.787043 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 04:33:47.787207 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1348,"thread_start_us":1536,"threads_started":1}
I20260619 04:33:47.787590 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:47.777668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43783 (local address 127.0.0.1:39788)
0619 04:33:47.778608 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:47.778624 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:47.778724 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:47.779442 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:47.779449 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:47.779474 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:47.779883 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:47.779901 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.782414 (+ 2513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.782419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:47.784804 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:47.784818 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.786112 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.786118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:47.786331 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:47.786382 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 04:33:47.786489 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 04:33:47.786687 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":108,"thread_start_us":795,"threads_started":1}
I20260619 04:33:47.787770 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:47.778594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40585 (local address 127.0.0.1:60482)
0619 04:33:47.779189 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:47.779206 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:47.779301 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:47.780682 (+ 1381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:47.780686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:47.780709 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:47.781073 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:47.781084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.782821 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.782826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:47.784956 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:47.784967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.786938 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.786944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:47.787118 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:47.787148 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 04:33:47.787226 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 04:33:47.787351 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":3,"thread_start_us":761,"threads_started":1}
I20260619 04:33:47.789250 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:47.779057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60482 (local address 127.0.97.1:40585)
0619 04:33:47.779993 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:47.780001 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:47.780024 (+ 23us) server_negotiation.cc:407] Connection header received
0619 04:33:47.780106 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:47.780111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:47.780257 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:47.780424 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:47.781256 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.782691 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.785095 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.786498 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.786625 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:47.786805 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:47.788754 (+ 1949us) server_negotiation.cc:299] Negotiation successful
0619 04:33:47.788919 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":696,"threads_started":1}
I20260619 04:33:47.791505 4774 tablet_service.cc:1467] Processing CreateTablet for tablet 10e03b297d3d4bc9906871ef61df4db6 (DEFAULT_TABLE table=test-table [id=c50dc8cf36f449f4b8f061dbacfa56c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:47.791505 4906 tablet_service.cc:1467] Processing CreateTablet for tablet 10e03b297d3d4bc9906871ef61df4db6 (DEFAULT_TABLE table=test-table [id=c50dc8cf36f449f4b8f061dbacfa56c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:47.792160 4641 tablet_service.cc:1467] Processing CreateTablet for tablet 10e03b297d3d4bc9906871ef61df4db6 (DEFAULT_TABLE table=test-table [id=c50dc8cf36f449f4b8f061dbacfa56c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:47.792923 4906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e03b297d3d4bc9906871ef61df4db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:47.792924 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e03b297d3d4bc9906871ef61df4db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:47.793984 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e03b297d3d4bc9906871ef61df4db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:47.809078 4990 tablet_bootstrap.cc:492] T 10e03b297d3d4bc9906871ef61df4db6 P 326635b3942e44dc9c32a73d520a0424: Bootstrap starting.
I20260619 04:33:47.807121 4989 tablet_bootstrap.cc:492] T 10e03b297d3d4bc9906871ef61df4db6 P 368762b3618e4991b852b858fcb1bc2c: Bootstrap starting.
I20260619 04:33:47.811841 4989 tablet_bootstrap.cc:654] T 10e03b297d3d4bc9906871ef61df4db6 P 368762b3618e4991b852b858fcb1bc2c: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:47.812610 4990 tablet_bootstrap.cc:654] T 10e03b297d3d4bc9906871ef61df4db6 P 326635b3942e44dc9c32a73d520a0424: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:47.812675 4991 tablet_bootstrap.cc:492] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a: Bootstrap starting.
I20260619 04:33:47.813109 4989 log.cc:824] T 10e03b297d3d4bc9906871ef61df4db6 P 368762b3618e4991b852b858fcb1bc2c: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:47.814033 4990 log.cc:824] T 10e03b297d3d4bc9906871ef61df4db6 P 326635b3942e44dc9c32a73d520a0424: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:47.815835 4991 tablet_bootstrap.cc:654] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:47.816162 4989 tablet_bootstrap.cc:492] T 10e03b297d3d4bc9906871ef61df4db6 P 368762b3618e4991b852b858fcb1bc2c: No bootstrap required, opened a new log
I20260619 04:33:47.816502 4989 ts_tablet_manager.cc:1397] T 10e03b297d3d4bc9906871ef61df4db6 P 368762b3618e4991b852b858fcb1bc2c: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 04:33:47.817021 4991 log.cc:824] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:47.817139 4990 tablet_bootstrap.cc:492] T 10e03b297d3d4bc9906871ef61df4db6 P 326635b3942e44dc9c32a73d520a0424: No bootstrap required, opened a new log
I20260619 04:33:47.817382 4990 ts_tablet_manager.cc:1397] T 10e03b297d3d4bc9906871ef61df4db6 P 326635b3942e44dc9c32a73d520a0424: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260619 04:33:47.820662 4706 heartbeater.cc:503] Master 127.0.97.62:42619 was elected leader, sending a full tablet report...
I20260619 04:33:47.825393 4990 raft_consensus.cc:348] T 10e03b297d3d4bc9906871ef61df4db6 P 326635b3942e44dc9c32a73d520a0424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "368762b3618e4991b852b858fcb1bc2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } } peers { permanent_uuid: "2538c7734fc44e6ea2c8daed2e49972a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40585 } } peers { permanent_uuid: "326635b3942e44dc9c32a73d520a0424" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43783 } }
I20260619 04:33:47.825970 4990 raft_consensus.cc:374] T 10e03b297d3d4bc9906871ef61df4db6 P 326635b3942e44dc9c32a73d520a0424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:47.826114 4990 raft_consensus.cc:729] T 10e03b297d3d4bc9906871ef61df4db6 P 326635b3942e44dc9c32a73d520a0424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 326635b3942e44dc9c32a73d520a0424, State: Initialized, Role: FOLLOWER
I20260619 04:33:47.827011 4990 consensus_queue.cc:260] T 10e03b297d3d4bc9906871ef61df4db6 P 326635b3942e44dc9c32a73d520a0424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "368762b3618e4991b852b858fcb1bc2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } } peers { permanent_uuid: "2538c7734fc44e6ea2c8daed2e49972a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40585 } } peers { permanent_uuid: "326635b3942e44dc9c32a73d520a0424" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43783 } }
I20260619 04:33:47.827143 4989 raft_consensus.cc:348] T 10e03b297d3d4bc9906871ef61df4db6 P 368762b3618e4991b852b858fcb1bc2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "368762b3618e4991b852b858fcb1bc2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } } peers { permanent_uuid: "2538c7734fc44e6ea2c8daed2e49972a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40585 } } peers { permanent_uuid: "326635b3942e44dc9c32a73d520a0424" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43783 } }
I20260619 04:33:47.827656 4989 raft_consensus.cc:374] T 10e03b297d3d4bc9906871ef61df4db6 P 368762b3618e4991b852b858fcb1bc2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:47.827756 4989 raft_consensus.cc:729] T 10e03b297d3d4bc9906871ef61df4db6 P 368762b3618e4991b852b858fcb1bc2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 368762b3618e4991b852b858fcb1bc2c, State: Initialized, Role: FOLLOWER
I20260619 04:33:47.828276 4989 consensus_queue.cc:260] T 10e03b297d3d4bc9906871ef61df4db6 P 368762b3618e4991b852b858fcb1bc2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "368762b3618e4991b852b858fcb1bc2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } } peers { permanent_uuid: "2538c7734fc44e6ea2c8daed2e49972a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40585 } } peers { permanent_uuid: "326635b3942e44dc9c32a73d520a0424" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43783 } }
I20260619 04:33:47.828691 4991 tablet_bootstrap.cc:492] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a: No bootstrap required, opened a new log
I20260619 04:33:47.828995 4991 ts_tablet_manager.cc:1397] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
I20260619 04:33:47.829154 4971 heartbeater.cc:503] Master 127.0.97.62:42619 was elected leader, sending a full tablet report...
I20260619 04:33:47.829869 4990 ts_tablet_manager.cc:1428] T 10e03b297d3d4bc9906871ef61df4db6 P 326635b3942e44dc9c32a73d520a0424: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260619 04:33:47.830801 4839 heartbeater.cc:503] Master 127.0.97.62:42619 was elected leader, sending a full tablet report...
I20260619 04:33:47.832302 4989 ts_tablet_manager.cc:1428] T 10e03b297d3d4bc9906871ef61df4db6 P 368762b3618e4991b852b858fcb1bc2c: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260619 04:33:47.837297 4991 raft_consensus.cc:348] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "368762b3618e4991b852b858fcb1bc2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } } peers { permanent_uuid: "2538c7734fc44e6ea2c8daed2e49972a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40585 } } peers { permanent_uuid: "326635b3942e44dc9c32a73d520a0424" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43783 } }
I20260619 04:33:47.837634 4991 raft_consensus.cc:374] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:47.837728 4991 raft_consensus.cc:729] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2538c7734fc44e6ea2c8daed2e49972a, State: Initialized, Role: FOLLOWER
I20260619 04:33:47.838248 4991 consensus_queue.cc:260] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "368762b3618e4991b852b858fcb1bc2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } } peers { permanent_uuid: "2538c7734fc44e6ea2c8daed2e49972a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40585 } } peers { permanent_uuid: "326635b3942e44dc9c32a73d520a0424" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43783 } }
I20260619 04:33:47.839591 4991 ts_tablet_manager.cc:1428] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260619 04:33:47.943643 4997 raft_consensus.cc:482] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:33:47.943914 4997 raft_consensus.cc:504] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "368762b3618e4991b852b858fcb1bc2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } } peers { permanent_uuid: "2538c7734fc44e6ea2c8daed2e49972a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40585 } } peers { permanent_uuid: "326635b3942e44dc9c32a73d520a0424" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43783 } }
I20260619 04:33:47.945590 4997 leader_election.cc:288] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 368762b3618e4991b852b858fcb1bc2c (127.0.97.2:42011), 326635b3942e44dc9c32a73d520a0424 (127.0.97.3:43783)
I20260619 04:33:47.951776 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:47.945904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54911 (local address 127.0.97.2:42011)
0619 04:33:47.946100 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:47.946105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:47.946810 (+ 705us) server_negotiation.cc:407] Connection header received
0619 04:33:47.947006 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:47.947010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:47.947068 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:47.947138 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:47.947705 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.948428 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.950160 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.950901 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.951016 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:47.951108 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:47.951399 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 04:33:47.951522 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 04:33:47.951779 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:47.946016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37985 (local address 127.0.97.3:43783)
0619 04:33:47.946215 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:47.946221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:47.946876 (+ 655us) server_negotiation.cc:407] Connection header received
0619 04:33:47.947047 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:47.947051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:47.947106 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:47.947189 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:33:47.947797 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.948489 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.950171 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.950917 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.951032 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:47.951126 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:47.951444 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 04:33:47.951559 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:33:47.951915 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:47.945762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42011 (local address 127.0.97.1:54911)
0619 04:33:47.946711 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:47.946730 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:47.946872 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:47.947159 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:47.947163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:47.947184 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:47.947578 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:47.947587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.948551 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.948556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:47.950025 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:47.950037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.951115 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.951121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:47.951253 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:47.951292 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 04:33:47.951365 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 04:33:47.951482 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":23,"thread_start_us":624,"threads_started":1}
I20260619 04:33:47.951967 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:47.945964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43783 (local address 127.0.97.1:37985)
0619 04:33:47.946798 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:47.946811 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:47.946910 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:47.947206 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:47.947210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:47.947234 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:33:47.947645 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:47.947657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.948620 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.948624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:47.950033 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:47.950044 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:47.951151 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:47.951158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:47.951311 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:47.951332 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 04:33:47.951406 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 04:33:47.951526 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":50,"thread_start_us":553,"threads_started":1}
I20260619 04:33:47.952811 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e03b297d3d4bc9906871ef61df4db6" candidate_uuid: "2538c7734fc44e6ea2c8daed2e49972a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "326635b3942e44dc9c32a73d520a0424" is_pre_election: true
I20260619 04:33:47.952822 4794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e03b297d3d4bc9906871ef61df4db6" candidate_uuid: "2538c7734fc44e6ea2c8daed2e49972a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "368762b3618e4991b852b858fcb1bc2c" is_pre_election: true
I20260619 04:33:47.953227 4926 raft_consensus.cc:2454] T 10e03b297d3d4bc9906871ef61df4db6 P 326635b3942e44dc9c32a73d520a0424 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2538c7734fc44e6ea2c8daed2e49972a in term 0.
I20260619 04:33:47.953240 4794 raft_consensus.cc:2454] T 10e03b297d3d4bc9906871ef61df4db6 P 368762b3618e4991b852b858fcb1bc2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2538c7734fc44e6ea2c8daed2e49972a in term 0.
I20260619 04:33:47.953783 4595 leader_election.cc:302] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2538c7734fc44e6ea2c8daed2e49972a, 326635b3942e44dc9c32a73d520a0424, 368762b3618e4991b852b858fcb1bc2c; no voters:
I20260619 04:33:47.954125 4997 raft_consensus.cc:2781] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:33:47.954255 4997 raft_consensus.cc:482] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:33:47.954341 4997 raft_consensus.cc:3037] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:47.956355 4997 raft_consensus.cc:504] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "368762b3618e4991b852b858fcb1bc2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } } peers { permanent_uuid: "2538c7734fc44e6ea2c8daed2e49972a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40585 } } peers { permanent_uuid: "326635b3942e44dc9c32a73d520a0424" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43783 } }
I20260619 04:33:47.956915 4997 leader_election.cc:288] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a [CANDIDATE]: Term 1 election: Requested vote from peers 368762b3618e4991b852b858fcb1bc2c (127.0.97.2:42011), 326635b3942e44dc9c32a73d520a0424 (127.0.97.3:43783)
I20260619 04:33:47.957288 4794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e03b297d3d4bc9906871ef61df4db6" candidate_uuid: "2538c7734fc44e6ea2c8daed2e49972a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "368762b3618e4991b852b858fcb1bc2c"
I20260619 04:33:47.957315 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e03b297d3d4bc9906871ef61df4db6" candidate_uuid: "2538c7734fc44e6ea2c8daed2e49972a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "326635b3942e44dc9c32a73d520a0424"
I20260619 04:33:47.957448 4794 raft_consensus.cc:3037] T 10e03b297d3d4bc9906871ef61df4db6 P 368762b3618e4991b852b858fcb1bc2c [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:47.957449 4926 raft_consensus.cc:3037] T 10e03b297d3d4bc9906871ef61df4db6 P 326635b3942e44dc9c32a73d520a0424 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:47.959399 4926 raft_consensus.cc:2454] T 10e03b297d3d4bc9906871ef61df4db6 P 326635b3942e44dc9c32a73d520a0424 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2538c7734fc44e6ea2c8daed2e49972a in term 1.
I20260619 04:33:47.959431 4794 raft_consensus.cc:2454] T 10e03b297d3d4bc9906871ef61df4db6 P 368762b3618e4991b852b858fcb1bc2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2538c7734fc44e6ea2c8daed2e49972a in term 1.
I20260619 04:33:47.959715 4595 leader_election.cc:302] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2538c7734fc44e6ea2c8daed2e49972a, 326635b3942e44dc9c32a73d520a0424; no voters:
I20260619 04:33:47.959946 4997 raft_consensus.cc:2781] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:47.960314 4997 raft_consensus.cc:686] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a [term 1 LEADER]: Becoming Leader. State: Replica: 2538c7734fc44e6ea2c8daed2e49972a, State: Running, Role: LEADER
I20260619 04:33:47.960752 4997 consensus_queue.cc:237] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "368762b3618e4991b852b858fcb1bc2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } } peers { permanent_uuid: "2538c7734fc44e6ea2c8daed2e49972a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40585 } } peers { permanent_uuid: "326635b3942e44dc9c32a73d520a0424" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43783 } }
W20260619 04:33:47.961539 4972 tablet.cc:2369] T 10e03b297d3d4bc9906871ef61df4db6 P 326635b3942e44dc9c32a73d520a0424: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:33:47.965035 4517 catalog_manager.cc:5495] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2538c7734fc44e6ea2c8daed2e49972a (127.0.97.1). New cstate: current_term: 1 leader_uuid: "2538c7734fc44e6ea2c8daed2e49972a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "368762b3618e4991b852b858fcb1bc2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2538c7734fc44e6ea2c8daed2e49972a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "326635b3942e44dc9c32a73d520a0424" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43783 } health_report { overall_health: UNKNOWN } } }
W20260619 04:33:48.002738 4840 tablet.cc:2369] T 10e03b297d3d4bc9906871ef61df4db6 P 368762b3618e4991b852b858fcb1bc2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:33:48.023432 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:48.018559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42506 (local address 127.0.97.1:40585)
0619 04:33:48.018762 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:48.018768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:48.018785 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:33:48.018872 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:48.018876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:48.018947 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:48.019101 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:33:48.019471 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:48.020153 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:48.021478 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:48.021865 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:48.021980 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:48.022409 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:33:48.022877 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:33:48.023026 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:48.023107 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 04:33:48.023182 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 04:33:48.031234 4926 raft_consensus.cc:1261] T 10e03b297d3d4bc9906871ef61df4db6 P 326635b3942e44dc9c32a73d520a0424 [term 1 FOLLOWER]: Refusing update from remote peer 2538c7734fc44e6ea2c8daed2e49972a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:33:48.031234 4794 raft_consensus.cc:1261] T 10e03b297d3d4bc9906871ef61df4db6 P 368762b3618e4991b852b858fcb1bc2c [term 1 FOLLOWER]: Refusing update from remote peer 2538c7734fc44e6ea2c8daed2e49972a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:33:48.032084 4997 consensus_queue.cc:1035] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a [LEADER]: Connected to new peer: Peer: permanent_uuid: "326635b3942e44dc9c32a73d520a0424" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:33:48.032343 5000 consensus_queue.cc:1035] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a [LEADER]: Connected to new peer: Peer: permanent_uuid: "368762b3618e4991b852b858fcb1bc2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:33:48.044073 5003 mvcc.cc:205] Tried to move back new op lower bound from 7298431500407164928 to 7298431500130574336. Current Snapshot: MvccSnapshot[applied={T|T < 7298431500407164928}]
I20260619 04:33:48.044646 5005 mvcc.cc:205] Tried to move back new op lower bound from 7298431500407164928 to 7298431500130574336. Current Snapshot: MvccSnapshot[applied={T|T < 7298431500407164928}]
I20260619 04:33:48.068661 4517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:33:48.068936 4517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:33:48.072165 4517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd163f9fed274d319d90f8ce3edbdbc3: Sending DeleteTablet for 3 replicas of tablet 10e03b297d3d4bc9906871ef61df4db6
I20260619 04:33:48.073143 4774 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e03b297d3d4bc9906871ef61df4db6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34434
I20260619 04:33:48.073274 4641 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e03b297d3d4bc9906871ef61df4db6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60482
I20260619 04:33:48.073709 4906 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e03b297d3d4bc9906871ef61df4db6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39788
I20260619 04:33:48.074981 5013 tablet_replica.cc:330] T 10e03b297d3d4bc9906871ef61df4db6 P 368762b3618e4991b852b858fcb1bc2c: stopping tablet replica
I20260619 04:33:48.075443 5013 raft_consensus.cc:2229] T 10e03b297d3d4bc9906871ef61df4db6 P 368762b3618e4991b852b858fcb1bc2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:33:48.075968 5013 raft_consensus.cc:2258] T 10e03b297d3d4bc9906871ef61df4db6 P 368762b3618e4991b852b858fcb1bc2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:48.075981 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 4579
I20260619 04:33:48.076732 5014 tablet_replica.cc:330] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a: stopping tablet replica
I20260619 04:33:48.077076 5014 raft_consensus.cc:2229] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:33:48.077543 5014 raft_consensus.cc:2258] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:48.078123 5013 ts_tablet_manager.cc:1905] T 10e03b297d3d4bc9906871ef61df4db6 P 368762b3618e4991b852b858fcb1bc2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:48.079370 5014 ts_tablet_manager.cc:1905] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:48.085302 5013 ts_tablet_manager.cc:1918] T 10e03b297d3d4bc9906871ef61df4db6 P 368762b3618e4991b852b858fcb1bc2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:33:48.085542 5013 log.cc:1196] T 10e03b297d3d4bc9906871ef61df4db6 P 368762b3618e4991b852b858fcb1bc2c: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/10e03b297d3d4bc9906871ef61df4db6
I20260619 04:33:48.086114 5013 ts_tablet_manager.cc:1939] T 10e03b297d3d4bc9906871ef61df4db6 P 368762b3618e4991b852b858fcb1bc2c: Deleting consensus metadata
I20260619 04:33:48.086318 5014 ts_tablet_manager.cc:1918] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:33:48.086508 5014 log.cc:1196] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-0/wal/wals/10e03b297d3d4bc9906871ef61df4db6
I20260619 04:33:48.087047 5014 ts_tablet_manager.cc:1939] T 10e03b297d3d4bc9906871ef61df4db6 P 2538c7734fc44e6ea2c8daed2e49972a: Deleting consensus metadata
I20260619 04:33:48.088119 4505 catalog_manager.cc:4841] TS 368762b3618e4991b852b858fcb1bc2c (127.0.97.2:42011): tablet 10e03b297d3d4bc9906871ef61df4db6 (table test-table [id=c50dc8cf36f449f4b8f061dbacfa56c2]) successfully deleted
I20260619 04:33:48.088640 4505 catalog_manager.cc:4841] TS 2538c7734fc44e6ea2c8daed2e49972a (127.0.97.1:40585): tablet 10e03b297d3d4bc9906871ef61df4db6 (table test-table [id=c50dc8cf36f449f4b8f061dbacfa56c2]) successfully deleted
I20260619 04:33:48.088922 5015 tablet_replica.cc:330] T 10e03b297d3d4bc9906871ef61df4db6 P 326635b3942e44dc9c32a73d520a0424: stopping tablet replica
I20260619 04:33:48.089334 5015 raft_consensus.cc:2229] T 10e03b297d3d4bc9906871ef61df4db6 P 326635b3942e44dc9c32a73d520a0424 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:33:48.089807 5015 raft_consensus.cc:2258] T 10e03b297d3d4bc9906871ef61df4db6 P 326635b3942e44dc9c32a73d520a0424 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:48.091833 5015 ts_tablet_manager.cc:1905] T 10e03b297d3d4bc9906871ef61df4db6 P 326635b3942e44dc9c32a73d520a0424: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 04:33:48 dist-test-slave-n3tq krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 04:33:48.098549 5015 ts_tablet_manager.cc:1918] T 10e03b297d3d4bc9906871ef61df4db6 P 326635b3942e44dc9c32a73d520a0424: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:33:48.098721 5015 log.cc:1196] T 10e03b297d3d4bc9906871ef61df4db6 P 326635b3942e44dc9c32a73d520a0424: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/10e03b297d3d4bc9906871ef61df4db6
I20260619 04:33:48.099213 5015 ts_tablet_manager.cc:1939] T 10e03b297d3d4bc9906871ef61df4db6 P 326635b3942e44dc9c32a73d520a0424: Deleting consensus metadata
I20260619 04:33:48.100901 4503 catalog_manager.cc:4841] TS 326635b3942e44dc9c32a73d520a0424 (127.0.97.3:43783): tablet 10e03b297d3d4bc9906871ef61df4db6 (table test-table [id=c50dc8cf36f449f4b8f061dbacfa56c2]) successfully deleted
I20260619 04:33:48.102273 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:48.076641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42512 (local address 127.0.97.1:40585)
0619 04:33:48.076798 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:48.076805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:48.088794 (+ 11989us) server_negotiation.cc:407] Connection header received
0619 04:33:48.088847 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:48.088850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:48.088914 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:48.089001 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:48.090346 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:48.091175 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:48.092411 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:48.092863 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:48.092969 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:48.096599 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:48.096609 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:48.096623 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:48.096660 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:48.099591 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:48.100193 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:48.100201 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:48.100205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:48.100333 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:48.100726 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:48.100731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:48.100735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:48.101169 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:48.101484 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:48.101742 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 04:33:48.101943 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 04:33:48.102871 4701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:48.196885 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 4579
I20260619 04:33:48.210788 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 4712
Jun 19 04:33:48 dist-test-slave-n3tq krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 04:33:48.224061 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:48.211498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46838 (local address 127.0.97.2:42011)
0619 04:33:48.211665 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:48.211670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:48.211689 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:33:48.211796 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:48.211799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:48.211860 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:48.211951 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:48.213199 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:48.213956 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:48.215030 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:48.215393 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:48.215489 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:48.219245 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:48.219258 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:48.219274 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:48.219305 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:48.221819 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:48.222400 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:48.222408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:48.222410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:48.222537 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:48.222835 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:48.222838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:48.222840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:48.223171 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:48.223421 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:48.223593 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 04:33:48.223755 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 04:33:48.224702 4834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:48.310881 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 4712
I20260619 04:33:48.324076 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 4844
Jun 19 04:33:48 dist-test-slave-n3tq krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 04:33:48.337764 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:48.324803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45676 (local address 127.0.97.3:43783)
0619 04:33:48.324964 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:48.324969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:48.324988 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:33:48.325196 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:48.325201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:48.325265 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:48.325348 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:48.326534 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:48.327261 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:48.328426 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:48.328851 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:48.328958 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:48.332794 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:48.332808 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:48.332823 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:48.332858 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:48.335224 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:48.335791 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:48.335799 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:48.335801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:48.335912 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:48.336268 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:48.336271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:48.336274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:48.336730 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:48.336993 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:48.337272 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 04:33:48.337468 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 04:33:48.338351 4966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:48.423486 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 4844
I20260619 04:33:48.435845 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 4487
I20260619 04:33:48.436882 4548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:48.541035 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 4487
2026-06-19T04:33:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3849 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T04:33:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:33:48Z Disabled control of system clock
I20260619 04:33:48.598205 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40889
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 04:33:48.899016 5024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:48.899291 5024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:48.899395 5024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:48.906517 5024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:33:48.906630 5024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:48.906670 5024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:48.906747 5024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:33:48.906805 5024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:33:48.915694 5024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40889
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:48.917068 5024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:48.918776 5024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:48.930390 5030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:48.930543 5032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:48.930864 5029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:48.931380 5024 server_base.cc:1034] running on GCE node
I20260619 04:33:48.932104 5024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:48.933413 5024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:48.934612 5024 hybrid_clock.cc:648] HybridClock initialized: now 1781843628934556 us; error 44 us; skew 500 ppm
I20260619 04:33:48.934921 5024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:33:48.937129 5024 webserver.cc:460] Webserver started at http://127.0.97.62:38737/ using document root <none> and password file <none>
I20260619 04:33:48.937774 5024 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:48.937870 5024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:48.938169 5024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:48.940922 5024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "276ec85913c14fc2b0ee751e33df0a26"
format_stamp: "Formatted at 2026-06-19 04:33:48 on dist-test-slave-n3tq"
server_key: "611cd97bf1d76a836484b15e29470bd8"
server_key_iv: "2382b9dc21c21f7c207b620ac50b8180"
server_key_version: "encryptionkey@0"
I20260619 04:33:48.941648 5024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "276ec85913c14fc2b0ee751e33df0a26"
format_stamp: "Formatted at 2026-06-19 04:33:48 on dist-test-slave-n3tq"
server_key: "611cd97bf1d76a836484b15e29470bd8"
server_key_iv: "2382b9dc21c21f7c207b620ac50b8180"
server_key_version: "encryptionkey@0"
I20260619 04:33:48.946352 5024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 04:33:48.949445 5038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:48.950649 5024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 04:33:48.950800 5024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "276ec85913c14fc2b0ee751e33df0a26"
format_stamp: "Formatted at 2026-06-19 04:33:48 on dist-test-slave-n3tq"
server_key: "611cd97bf1d76a836484b15e29470bd8"
server_key_iv: "2382b9dc21c21f7c207b620ac50b8180"
server_key_version: "encryptionkey@0"
I20260619 04:33:48.950979 5024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:48.956257 5024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:48.956986 5024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:48.957237 5024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:48.973253 5024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40889
I20260619 04:33:48.973284 5089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40889 every 8 connection(s)
I20260619 04:33:48.974450 5024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:33:48.978129 5090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:48.984175 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 5024
I20260619 04:33:48.984349 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:33:48.984947 388 external_mini_cluster.cc:1428] Setting key 4b36f351dbfd40a94eae9b74036d21f2
I20260619 04:33:48.988221 5090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26: Bootstrap starting.
I20260619 04:33:48.991452 5090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:48.992574 5090 log.cc:824] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:48.992861 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:48.986881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37024 (local address 127.0.97.62:40889)
0619 04:33:48.987801 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:48.987812 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:48.987853 (+ 41us) server_negotiation.cc:407] Connection header received
0619 04:33:48.989454 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:48.989486 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:48.989910 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:48.990221 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:48.991240 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:48.991246 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:48.991258 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:33:48.991272 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:48.991327 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:33:48.991345 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:48.991486 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:48.991667 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 04:33:48.991933 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":289,"threads_started":1}
I20260619 04:33:48.995303 5090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26: No bootstrap required, opened a new log
I20260619 04:33:49.001952 5090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276ec85913c14fc2b0ee751e33df0a26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40889 } }
I20260619 04:33:49.002314 5090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:49.002404 5090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 276ec85913c14fc2b0ee751e33df0a26, State: Initialized, Role: FOLLOWER
I20260619 04:33:49.002919 5090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276ec85913c14fc2b0ee751e33df0a26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40889 } }
I20260619 04:33:49.003136 5090 raft_consensus.cc:388] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:33:49.003248 5090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:33:49.003387 5090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:49.005513 5090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276ec85913c14fc2b0ee751e33df0a26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40889 } }
I20260619 04:33:49.005944 5090 leader_election.cc:302] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 276ec85913c14fc2b0ee751e33df0a26; no voters:
I20260619 04:33:49.006541 5090 leader_election.cc:288] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:33:49.006664 5095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:49.007283 5095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26 [term 1 LEADER]: Becoming Leader. State: Replica: 276ec85913c14fc2b0ee751e33df0a26, State: Running, Role: LEADER
I20260619 04:33:49.007860 5095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276ec85913c14fc2b0ee751e33df0a26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40889 } }
I20260619 04:33:49.008770 5090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:33:49.010699 5097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 276ec85913c14fc2b0ee751e33df0a26. Latest consensus state: current_term: 1 leader_uuid: "276ec85913c14fc2b0ee751e33df0a26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276ec85913c14fc2b0ee751e33df0a26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40889 } } }
I20260619 04:33:49.010673 5096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "276ec85913c14fc2b0ee751e33df0a26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276ec85913c14fc2b0ee751e33df0a26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40889 } } }
I20260619 04:33:49.011094 5097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26 [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:49.011098 5096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26 [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:49.012270 5101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:33:49.017225 5101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:33:49.026202 5101 catalog_manager.cc:1347] Generated new cluster ID: 5f6a41263a804213a4ac98bda6e483a7
I20260619 04:33:49.026309 5101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:33:49.037575 5101 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:33:49.037957 5101 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:33:49.046373 5101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26: Generated new TSK 0
I20260619 04:33:49.047137 5101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:33:49.058218 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40889
--builtin_ntp_servers=127.0.97.20:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 04:33:49.395087 5114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:49.395363 5114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:49.395449 5114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:49.402693 5114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:49.402786 5114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:49.402892 5114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:33:49.411634 5114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:49.412775 5114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:49.414419 5114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:49.426077 5119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:49.426141 5120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:49.426483 5122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:49.426916 5114 server_base.cc:1034] running on GCE node
I20260619 04:33:49.427420 5114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:49.428233 5114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:49.429414 5114 hybrid_clock.cc:648] HybridClock initialized: now 1781843629429361 us; error 49 us; skew 500 ppm
I20260619 04:33:49.429697 5114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:33:49.431677 5114 webserver.cc:460] Webserver started at http://127.0.97.1:35839/ using document root <none> and password file <none>
I20260619 04:33:49.432276 5114 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:49.432360 5114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:49.432674 5114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:49.435236 5114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "873cc61681de413cbcbadff4ca36c658"
format_stamp: "Formatted at 2026-06-19 04:33:49 on dist-test-slave-n3tq"
server_key: "747ec66dc34d2f95366443ef658f9e1e"
server_key_iv: "c615c10f5195f522e8bd75c08080d2fe"
server_key_version: "encryptionkey@0"
I20260619 04:33:49.435940 5114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "873cc61681de413cbcbadff4ca36c658"
format_stamp: "Formatted at 2026-06-19 04:33:49 on dist-test-slave-n3tq"
server_key: "747ec66dc34d2f95366443ef658f9e1e"
server_key_iv: "c615c10f5195f522e8bd75c08080d2fe"
server_key_version: "encryptionkey@0"
I20260619 04:33:49.440438 5114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260619 04:33:49.443480 5128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:49.444698 5114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260619 04:33:49.444828 5114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "873cc61681de413cbcbadff4ca36c658"
format_stamp: "Formatted at 2026-06-19 04:33:49 on dist-test-slave-n3tq"
server_key: "747ec66dc34d2f95366443ef658f9e1e"
server_key_iv: "c615c10f5195f522e8bd75c08080d2fe"
server_key_version: "encryptionkey@0"
I20260619 04:33:49.445012 5114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:49.450040 5114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:49.450722 5114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:49.450937 5114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:49.451722 5114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:49.452966 5114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:49.453054 5114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:49.453153 5114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:49.453225 5114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:49.480831 5114 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43487
I20260619 04:33:49.480880 5240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43487 every 8 connection(s)
I20260619 04:33:49.482064 5114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:33:49.488574 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 5114
I20260619 04:33:49.488783 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:33:49.489339 388 external_mini_cluster.cc:1428] Setting key 5e54ec47e96705bf1c4e69c54fa5b434
I20260619 04:33:49.489784 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:49.484510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41259 (local address 127.0.97.62:40889)
0619 04:33:49.484690 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:49.484696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:49.486137 (+ 1441us) server_negotiation.cc:407] Connection header received
0619 04:33:49.487717 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:49.487721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:49.487774 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:49.487867 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:49.489125 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:49.489130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:49.489135 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:33:49.489153 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:49.489183 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:33:49.489194 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:49.489303 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:49.489465 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 04:33:49.489528 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 04:33:49.490778 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:49.485136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40889 (local address 127.0.97.1:41259)
0619 04:33:49.485995 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:49.486034 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:49.487462 (+ 1428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:49.488039 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:49.488047 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:49.488152 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:49.488833 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:33:49.488836 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:49.488883 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:33:49.488974 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:49.489275 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:49.489285 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:49.489292 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 04:33:49.489540 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 04:33:49.489779 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":256,"threads_started":1}
I20260619 04:33:49.492170 5241 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40889
I20260619 04:33:49.492350 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40889
--builtin_ntp_servers=127.0.97.20:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 04:33:49.492442 5241 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:49.493003 5241 heartbeater.cc:511] Master 127.0.97.62:40889 requested a full tablet report, sending...
I20260619 04:33:49.495064 5055 ts_manager.cc:194] Registered new tserver with Master: 873cc61681de413cbcbadff4ca36c658 (127.0.97.1:43487)
W20260619 04:33:49.801397 5244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:49.801664 5244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:49.801751 5244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:49.808746 5244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:49.808840 5244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:49.808944 5244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:33:49.817828 5244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:49.819015 5244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:49.820708 5244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:49.833170 5250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:49.833202 5252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:49.833415 5249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:49.833933 5244 server_base.cc:1034] running on GCE node
I20260619 04:33:49.834475 5244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:49.835326 5244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:49.836555 5244 hybrid_clock.cc:648] HybridClock initialized: now 1781843629836503 us; error 37 us; skew 500 ppm
I20260619 04:33:49.836864 5244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:33:49.838968 5244 webserver.cc:460] Webserver started at http://127.0.97.2:46683/ using document root <none> and password file <none>
I20260619 04:33:49.839635 5244 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:49.839725 5244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:49.840013 5244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:49.842720 5244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "5d19408bd2f24b76a6584465d14dbc69"
format_stamp: "Formatted at 2026-06-19 04:33:49 on dist-test-slave-n3tq"
server_key: "293e4b2630e35bf9d2cd8b90adc446ce"
server_key_iv: "3d24458a74fa359e2223bcef2558d6d7"
server_key_version: "encryptionkey@0"
I20260619 04:33:49.843446 5244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5d19408bd2f24b76a6584465d14dbc69"
format_stamp: "Formatted at 2026-06-19 04:33:49 on dist-test-slave-n3tq"
server_key: "293e4b2630e35bf9d2cd8b90adc446ce"
server_key_iv: "3d24458a74fa359e2223bcef2558d6d7"
server_key_version: "encryptionkey@0"
I20260619 04:33:49.848148 5244 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 04:33:49.851280 5258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:49.852556 5244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 04:33:49.852695 5244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "5d19408bd2f24b76a6584465d14dbc69"
format_stamp: "Formatted at 2026-06-19 04:33:49 on dist-test-slave-n3tq"
server_key: "293e4b2630e35bf9d2cd8b90adc446ce"
server_key_iv: "3d24458a74fa359e2223bcef2558d6d7"
server_key_version: "encryptionkey@0"
I20260619 04:33:49.852891 5244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:49.858139 5244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:49.858844 5244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:49.859077 5244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:49.859898 5244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:49.861235 5244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:49.861325 5244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:49.861425 5244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:49.861493 5244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:49.889606 5244 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41555
I20260619 04:33:49.889669 5370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41555 every 8 connection(s)
I20260619 04:33:49.890832 5244 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:33:49.898330 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:49.893240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54165 (local address 127.0.97.62:40889)
0619 04:33:49.893403 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:49.893407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:49.894743 (+ 1336us) server_negotiation.cc:407] Connection header received
0619 04:33:49.896356 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:49.896362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:49.896425 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:49.896540 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:49.897664 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:49.897670 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:49.897676 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:33:49.897692 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:49.897719 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:33:49.897730 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:49.897839 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:49.897996 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 04:33:49.898057 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 04:33:49.899158 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:49.893733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40889 (local address 127.0.97.2:54165)
0619 04:33:49.894589 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:49.894623 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:49.896104 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:49.896752 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:49.896759 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:49.896848 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:49.897402 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:33:49.897404 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:49.897437 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:33:49.897514 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:49.897808 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:49.897818 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:49.897826 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 04:33:49.898065 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 04:33:49.898298 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":266,"threads_started":1}
I20260619 04:33:49.899214 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 5244
I20260619 04:33:49.899390 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:33:49.899860 388 external_mini_cluster.cc:1428] Setting key 0314610c1ac971d3f8e7a1ba87ee6ce4
I20260619 04:33:49.900440 5371 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40889
I20260619 04:33:49.900738 5371 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:49.901378 5371 heartbeater.cc:511] Master 127.0.97.62:40889 requested a full tablet report, sending...
I20260619 04:33:49.902511 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40889
--builtin_ntp_servers=127.0.97.20:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 04:33:49.902652 5055 ts_manager.cc:194] Registered new tserver with Master: 5d19408bd2f24b76a6584465d14dbc69 (127.0.97.2:41555)
W20260619 04:33:50.208691 5374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:50.209022 5374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:50.209173 5374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:50.216341 5374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:50.216434 5374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:50.216607 5374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:33:50.225490 5374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:50.226673 5374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:50.228327 5374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:50.239996 5382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:50.240077 5379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:50.240034 5380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:50.240860 5374 server_base.cc:1034] running on GCE node
I20260619 04:33:50.241402 5374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:50.242240 5374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:50.243424 5374 hybrid_clock.cc:648] HybridClock initialized: now 1781843630243380 us; error 37 us; skew 500 ppm
I20260619 04:33:50.243697 5374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:33:50.245685 5374 webserver.cc:460] Webserver started at http://127.0.97.3:41209/ using document root <none> and password file <none>
I20260619 04:33:50.246294 5374 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:50.246385 5374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:50.246654 5374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:50.249317 5374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "da8bc6f2729a4c7dbcfb709d2c27e8db"
format_stamp: "Formatted at 2026-06-19 04:33:50 on dist-test-slave-n3tq"
server_key: "8b5c56cca674d0c482373b42f8fffc79"
server_key_iv: "3d0f4c19cadc658cb09f83d32d4d3f3b"
server_key_version: "encryptionkey@0"
I20260619 04:33:50.250008 5374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "da8bc6f2729a4c7dbcfb709d2c27e8db"
format_stamp: "Formatted at 2026-06-19 04:33:50 on dist-test-slave-n3tq"
server_key: "8b5c56cca674d0c482373b42f8fffc79"
server_key_iv: "3d0f4c19cadc658cb09f83d32d4d3f3b"
server_key_version: "encryptionkey@0"
I20260619 04:33:50.254887 5374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:33:50.258052 5388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:50.259248 5374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 04:33:50.259385 5374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "da8bc6f2729a4c7dbcfb709d2c27e8db"
format_stamp: "Formatted at 2026-06-19 04:33:50 on dist-test-slave-n3tq"
server_key: "8b5c56cca674d0c482373b42f8fffc79"
server_key_iv: "3d0f4c19cadc658cb09f83d32d4d3f3b"
server_key_version: "encryptionkey@0"
I20260619 04:33:50.259572 5374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:50.264696 5374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:50.265385 5374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:50.265604 5374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:50.266443 5374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:50.267697 5374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:50.267786 5374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:50.267884 5374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:50.267949 5374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:50.296689 5374 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45341
I20260619 04:33:50.296732 5500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45341 every 8 connection(s)
I20260619 04:33:50.297933 5374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:33:50.299322 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 5374
I20260619 04:33:50.299500 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:33:50.299986 388 external_mini_cluster.cc:1428] Setting key a1767ce68c5efaeea81d1168d2d5d653
I20260619 04:33:50.306365 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:50.301173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53007 (local address 127.0.97.62:40889)
0619 04:33:50.301354 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:50.301359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:50.302900 (+ 1541us) server_negotiation.cc:407] Connection header received
0619 04:33:50.304570 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:50.304574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:50.304623 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:50.304695 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:50.305860 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:50.305863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:50.305866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:33:50.305878 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:50.305900 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:33:50.305909 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:50.305984 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:50.306143 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 04:33:50.306185 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 04:33:50.307235 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:50.301791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40889 (local address 127.0.97.3:53007)
0619 04:33:50.302757 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:50.302804 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:50.304316 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:50.304878 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:50.304886 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:50.304983 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:50.305628 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:33:50.305630 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:50.305664 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:33:50.305743 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:50.305972 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:50.305978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:50.305983 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 04:33:50.306191 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 04:33:50.306369 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":322,"threads_started":1}
I20260619 04:33:50.308393 5501 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40889
I20260619 04:33:50.308694 5501 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:50.309223 5501 heartbeater.cc:511] Master 127.0.97.62:40889 requested a full tablet report, sending...
I20260619 04:33:50.310487 5055 ts_manager.cc:194] Registered new tserver with Master: da8bc6f2729a4c7dbcfb709d2c27e8db (127.0.97.3:45341)
I20260619 04:33:50.314951 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:33:50.321328 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:50.319061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37026 (local address 127.0.97.62:40889)
0619 04:33:50.319211 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:50.319215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:50.319653 (+ 438us) server_negotiation.cc:407] Connection header received
0619 04:33:50.319802 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:50.319806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:50.319850 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:50.319923 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:50.320767 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:50.320773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:50.320778 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:33:50.320795 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:50.320822 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:33:50.320833 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:50.320932 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:50.321018 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 04:33:50.321070 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:33:50.324659 5055 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:33:50.330924 5055 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:33:50.358726 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:50.341760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43487 (local address 127.0.0.1:45684)
0619 04:33:50.351269 (+ 9509us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:50.351289 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:50.351392 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:50.356104 (+ 4712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:50.356112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:50.356216 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:50.357380 (+ 1164us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:33:50.357383 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:50.357410 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:33:50.357495 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:50.357995 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:50.358005 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:50.358016 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 04:33:50.358176 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 04:33:50.358256 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9355,"thread_start_us":9206,"threads_started":1}
I20260619 04:33:50.359140 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:50.342309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.0.97.1:43487)
0619 04:33:50.354244 (+ 11935us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:50.354253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:50.354289 (+ 36us) server_negotiation.cc:407] Connection header received
0619 04:33:50.354366 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:50.354375 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:50.354654 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:50.354791 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:50.357705 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:50.357713 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:50.357728 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:33:50.357752 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:50.357806 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:33:50.357826 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:50.357945 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:50.358732 (+ 787us) server_negotiation.cc:299] Negotiation successful
0619 04:33:50.358832 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11763,"thread_start_us":11452,"threads_started":1}
I20260619 04:33:50.360623 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:50.351603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45341 (local address 127.0.0.1:32938)
0619 04:33:50.357816 (+ 6213us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:50.357832 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:50.357929 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:50.359020 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:50.359025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:50.359045 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:50.359591 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:33:50.359594 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:50.359609 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:33:50.359646 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:50.360021 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:50.360025 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:50.360030 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 04:33:50.360140 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 04:33:50.360185 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6089,"thread_start_us":5950,"threads_started":1}
I20260619 04:33:50.360970 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:50.351650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32938 (local address 127.0.97.3:45341)
0619 04:33:50.357589 (+ 5939us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:50.357608 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:50.358047 (+ 439us) server_negotiation.cc:407] Connection header received
0619 04:33:50.358122 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:50.358128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:50.358337 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:50.358471 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:50.359804 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:50.359812 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:50.359825 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:33:50.359843 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:50.359896 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:33:50.359920 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:50.360081 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:50.360153 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 04:33:50.360250 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5770,"thread_start_us":5597,"threads_started":1}
I20260619 04:33:50.361027 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:50.341722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41555 (local address 127.0.0.1:46556)
0619 04:33:50.349359 (+ 7637us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:50.349406 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:50.349545 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:50.357724 (+ 8179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:50.357729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:50.357751 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:50.358450 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:33:50.358453 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:50.358469 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:33:50.358513 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:50.360218 (+ 1705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:50.360223 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:50.360228 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 04:33:50.360344 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 04:33:50.360391 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6747,"thread_start_us":6571,"threads_started":1}
I20260619 04:33:50.361698 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:50.341958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46556 (local address 127.0.97.2:41555)
0619 04:33:50.356053 (+ 14095us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:50.356062 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:50.356097 (+ 35us) server_negotiation.cc:407] Connection header received
0619 04:33:50.356167 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:50.356175 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:50.356395 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:50.358680 (+ 2285us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:50.359999 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:50.360006 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:50.360017 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:33:50.360033 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:50.360078 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:33:50.360100 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:50.361242 (+ 1142us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:50.361303 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 04:33:50.361393 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13894,"thread_start_us":13778,"threads_started":1}
I20260619 04:33:50.363060 5176 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed05ec9d1a64c18b9b8061f536ca3f0 (DEFAULT_TABLE table=test-table [id=b766d352706a435b9346d2973e82a3c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:50.364574 5176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed05ec9d1a64c18b9b8061f536ca3f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:50.364828 5306 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed05ec9d1a64c18b9b8061f536ca3f0 (DEFAULT_TABLE table=test-table [id=b766d352706a435b9346d2973e82a3c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:50.365090 5436 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed05ec9d1a64c18b9b8061f536ca3f0 (DEFAULT_TABLE table=test-table [id=b766d352706a435b9346d2973e82a3c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:33:50.365859 5306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed05ec9d1a64c18b9b8061f536ca3f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:50.366326 5436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed05ec9d1a64c18b9b8061f536ca3f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:50.376289 5519 tablet_bootstrap.cc:492] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 5d19408bd2f24b76a6584465d14dbc69: Bootstrap starting.
I20260619 04:33:50.378897 5520 tablet_bootstrap.cc:492] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658: Bootstrap starting.
I20260619 04:33:50.380179 5521 tablet_bootstrap.cc:492] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P da8bc6f2729a4c7dbcfb709d2c27e8db: Bootstrap starting.
I20260619 04:33:50.380720 5519 tablet_bootstrap.cc:654] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 5d19408bd2f24b76a6584465d14dbc69: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:50.381922 5519 log.cc:824] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 5d19408bd2f24b76a6584465d14dbc69: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:50.382225 5520 tablet_bootstrap.cc:654] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:50.382925 5521 tablet_bootstrap.cc:654] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P da8bc6f2729a4c7dbcfb709d2c27e8db: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:50.383451 5520 log.cc:824] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:50.383984 5521 log.cc:824] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P da8bc6f2729a4c7dbcfb709d2c27e8db: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:50.386790 5521 tablet_bootstrap.cc:492] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P da8bc6f2729a4c7dbcfb709d2c27e8db: No bootstrap required, opened a new log
I20260619 04:33:50.386960 5520 tablet_bootstrap.cc:492] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658: No bootstrap required, opened a new log
I20260619 04:33:50.386960 5519 tablet_bootstrap.cc:492] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 5d19408bd2f24b76a6584465d14dbc69: No bootstrap required, opened a new log
I20260619 04:33:50.387023 5521 ts_tablet_manager.cc:1397] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P da8bc6f2729a4c7dbcfb709d2c27e8db: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 04:33:50.387207 5519 ts_tablet_manager.cc:1397] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 5d19408bd2f24b76a6584465d14dbc69: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20260619 04:33:50.387210 5520 ts_tablet_manager.cc:1397] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260619 04:33:50.394075 5521 raft_consensus.cc:348] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P da8bc6f2729a4c7dbcfb709d2c27e8db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d19408bd2f24b76a6584465d14dbc69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41555 } } peers { permanent_uuid: "873cc61681de413cbcbadff4ca36c658" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43487 } } peers { permanent_uuid: "da8bc6f2729a4c7dbcfb709d2c27e8db" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45341 } }
I20260619 04:33:50.394420 5521 raft_consensus.cc:374] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P da8bc6f2729a4c7dbcfb709d2c27e8db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:50.394523 5521 raft_consensus.cc:729] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P da8bc6f2729a4c7dbcfb709d2c27e8db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da8bc6f2729a4c7dbcfb709d2c27e8db, State: Initialized, Role: FOLLOWER
I20260619 04:33:50.395067 5521 consensus_queue.cc:260] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P da8bc6f2729a4c7dbcfb709d2c27e8db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d19408bd2f24b76a6584465d14dbc69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41555 } } peers { permanent_uuid: "873cc61681de413cbcbadff4ca36c658" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43487 } } peers { permanent_uuid: "da8bc6f2729a4c7dbcfb709d2c27e8db" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45341 } }
I20260619 04:33:50.396081 5501 heartbeater.cc:503] Master 127.0.97.62:40889 was elected leader, sending a full tablet report...
I20260619 04:33:50.396761 5521 ts_tablet_manager.cc:1428] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P da8bc6f2729a4c7dbcfb709d2c27e8db: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260619 04:33:50.397172 5520 raft_consensus.cc:348] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d19408bd2f24b76a6584465d14dbc69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41555 } } peers { permanent_uuid: "873cc61681de413cbcbadff4ca36c658" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43487 } } peers { permanent_uuid: "da8bc6f2729a4c7dbcfb709d2c27e8db" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45341 } }
I20260619 04:33:50.397161 5519 raft_consensus.cc:348] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 5d19408bd2f24b76a6584465d14dbc69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d19408bd2f24b76a6584465d14dbc69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41555 } } peers { permanent_uuid: "873cc61681de413cbcbadff4ca36c658" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43487 } } peers { permanent_uuid: "da8bc6f2729a4c7dbcfb709d2c27e8db" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45341 } }
I20260619 04:33:50.397562 5519 raft_consensus.cc:374] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 5d19408bd2f24b76a6584465d14dbc69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:50.397562 5520 raft_consensus.cc:374] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:50.397653 5520 raft_consensus.cc:729] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873cc61681de413cbcbadff4ca36c658, State: Initialized, Role: FOLLOWER
I20260619 04:33:50.397665 5519 raft_consensus.cc:729] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 5d19408bd2f24b76a6584465d14dbc69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d19408bd2f24b76a6584465d14dbc69, State: Initialized, Role: FOLLOWER
I20260619 04:33:50.398247 5519 consensus_queue.cc:260] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 5d19408bd2f24b76a6584465d14dbc69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d19408bd2f24b76a6584465d14dbc69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41555 } } peers { permanent_uuid: "873cc61681de413cbcbadff4ca36c658" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43487 } } peers { permanent_uuid: "da8bc6f2729a4c7dbcfb709d2c27e8db" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45341 } }
I20260619 04:33:50.398247 5520 consensus_queue.cc:260] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d19408bd2f24b76a6584465d14dbc69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41555 } } peers { permanent_uuid: "873cc61681de413cbcbadff4ca36c658" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43487 } } peers { permanent_uuid: "da8bc6f2729a4c7dbcfb709d2c27e8db" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45341 } }
I20260619 04:33:50.400103 5519 ts_tablet_manager.cc:1428] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 5d19408bd2f24b76a6584465d14dbc69: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260619 04:33:50.400557 5371 heartbeater.cc:503] Master 127.0.97.62:40889 was elected leader, sending a full tablet report...
I20260619 04:33:50.401868 5520 ts_tablet_manager.cc:1428] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658: Time spent starting tablet: real 0.015s user 0.010s sys 0.006s
I20260619 04:33:50.402603 5241 heartbeater.cc:503] Master 127.0.97.62:40889 was elected leader, sending a full tablet report...
W20260619 04:33:50.484036 5242 tablet.cc:2369] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:33:50.549525 5502 tablet.cc:2369] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P da8bc6f2729a4c7dbcfb709d2c27e8db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:33:50.562083 5526 raft_consensus.cc:482] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:33:50.562412 5526 raft_consensus.cc:504] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d19408bd2f24b76a6584465d14dbc69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41555 } } peers { permanent_uuid: "873cc61681de413cbcbadff4ca36c658" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43487 } } peers { permanent_uuid: "da8bc6f2729a4c7dbcfb709d2c27e8db" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45341 } }
I20260619 04:33:50.564009 5526 leader_election.cc:288] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d19408bd2f24b76a6584465d14dbc69 (127.0.97.2:41555), da8bc6f2729a4c7dbcfb709d2c27e8db (127.0.97.3:45341)
I20260619 04:33:50.567278 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:50.564518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56877 (local address 127.0.97.3:45341)
0619 04:33:50.564689 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:50.564695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:50.565316 (+ 621us) server_negotiation.cc:407] Connection header received
0619 04:33:50.565487 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:50.565492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:50.565569 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:50.565650 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:50.566714 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:50.566719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:50.566723 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:33:50.566741 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:50.566777 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:33:50.566791 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:50.566900 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:50.566977 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 04:33:50.567034 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:33:50.567278 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:50.564392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36865 (local address 127.0.97.2:41555)
0619 04:33:50.564586 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:50.564592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:50.565376 (+ 784us) server_negotiation.cc:407] Connection header received
0619 04:33:50.565466 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:50.565471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:50.565541 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:50.565641 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:50.566723 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:50.566728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:50.566732 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:33:50.566749 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:50.566778 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:33:50.566792 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:50.566900 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:50.566977 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 04:33:50.567058 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 04:33:50.567520 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:50.564297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45341 (local address 127.0.97.1:56877)
0619 04:33:50.565201 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:50.565223 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:50.565360 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:50.565688 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:50.565693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:50.565717 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:50.566545 (+ 828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:33:50.566548 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:50.566558 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:33:50.566585 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:50.566882 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:50.566887 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:50.566892 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 04:33:50.567046 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 04:33:50.567103 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":15,"thread_start_us":707,"threads_started":1}
I20260619 04:33:50.567783 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:50.564353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41555 (local address 127.0.97.1:36865)
0619 04:33:50.565278 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:50.565292 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:50.565369 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:50.565688 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:50.565693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:50.565717 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:50.566494 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:33:50.566498 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:50.566519 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:33:50.566566 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:50.566882 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:50.566887 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:50.566892 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 04:33:50.567024 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 04:33:50.567091 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":404,"threads_started":1}
I20260619 04:33:50.568486 5456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed05ec9d1a64c18b9b8061f536ca3f0" candidate_uuid: "873cc61681de413cbcbadff4ca36c658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da8bc6f2729a4c7dbcfb709d2c27e8db" is_pre_election: true
I20260619 04:33:50.568621 5326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed05ec9d1a64c18b9b8061f536ca3f0" candidate_uuid: "873cc61681de413cbcbadff4ca36c658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d19408bd2f24b76a6584465d14dbc69" is_pre_election: true
I20260619 04:33:50.568962 5456 raft_consensus.cc:2454] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P da8bc6f2729a4c7dbcfb709d2c27e8db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873cc61681de413cbcbadff4ca36c658 in term 0.
I20260619 04:33:50.569020 5326 raft_consensus.cc:2454] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 5d19408bd2f24b76a6584465d14dbc69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873cc61681de413cbcbadff4ca36c658 in term 0.
I20260619 04:33:50.569528 5129 leader_election.cc:302] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d19408bd2f24b76a6584465d14dbc69, 873cc61681de413cbcbadff4ca36c658; no voters:
I20260619 04:33:50.569871 5526 raft_consensus.cc:2781] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:33:50.570004 5526 raft_consensus.cc:482] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:33:50.570096 5526 raft_consensus.cc:3037] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:50.572082 5526 raft_consensus.cc:504] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d19408bd2f24b76a6584465d14dbc69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41555 } } peers { permanent_uuid: "873cc61681de413cbcbadff4ca36c658" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43487 } } peers { permanent_uuid: "da8bc6f2729a4c7dbcfb709d2c27e8db" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45341 } }
I20260619 04:33:50.572659 5526 leader_election.cc:288] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658 [CANDIDATE]: Term 1 election: Requested vote from peers 5d19408bd2f24b76a6584465d14dbc69 (127.0.97.2:41555), da8bc6f2729a4c7dbcfb709d2c27e8db (127.0.97.3:45341)
I20260619 04:33:50.573011 5326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed05ec9d1a64c18b9b8061f536ca3f0" candidate_uuid: "873cc61681de413cbcbadff4ca36c658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d19408bd2f24b76a6584465d14dbc69"
I20260619 04:33:50.573108 5456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed05ec9d1a64c18b9b8061f536ca3f0" candidate_uuid: "873cc61681de413cbcbadff4ca36c658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da8bc6f2729a4c7dbcfb709d2c27e8db"
I20260619 04:33:50.573172 5326 raft_consensus.cc:3037] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 5d19408bd2f24b76a6584465d14dbc69 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:50.573292 5456 raft_consensus.cc:3037] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P da8bc6f2729a4c7dbcfb709d2c27e8db [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:50.575209 5326 raft_consensus.cc:2454] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 5d19408bd2f24b76a6584465d14dbc69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873cc61681de413cbcbadff4ca36c658 in term 1.
I20260619 04:33:50.575325 5456 raft_consensus.cc:2454] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P da8bc6f2729a4c7dbcfb709d2c27e8db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873cc61681de413cbcbadff4ca36c658 in term 1.
I20260619 04:33:50.575523 5129 leader_election.cc:302] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d19408bd2f24b76a6584465d14dbc69, 873cc61681de413cbcbadff4ca36c658; no voters:
I20260619 04:33:50.575766 5526 raft_consensus.cc:2781] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:50.576184 5526 raft_consensus.cc:686] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658 [term 1 LEADER]: Becoming Leader. State: Replica: 873cc61681de413cbcbadff4ca36c658, State: Running, Role: LEADER
I20260619 04:33:50.576645 5526 consensus_queue.cc:237] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d19408bd2f24b76a6584465d14dbc69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41555 } } peers { permanent_uuid: "873cc61681de413cbcbadff4ca36c658" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43487 } } peers { permanent_uuid: "da8bc6f2729a4c7dbcfb709d2c27e8db" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45341 } }
I20260619 04:33:50.580397 5055 catalog_manager.cc:5495] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658 reported cstate change: term changed from 0 to 1, leader changed from <none> to 873cc61681de413cbcbadff4ca36c658 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "873cc61681de413cbcbadff4ca36c658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d19408bd2f24b76a6584465d14dbc69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "873cc61681de413cbcbadff4ca36c658" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da8bc6f2729a4c7dbcfb709d2c27e8db" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20260619 04:33:50.603297 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:50.601609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45686 (local address 127.0.97.1:43487)
0619 04:33:50.601761 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:50.601766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:50.601784 (+ 18us) server_negotiation.cc:407] Connection header received
0619 04:33:50.601927 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:50.601931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:50.602020 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:50.602101 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:50.602837 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:50.602840 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:50.602844 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:33:50.602856 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:50.602883 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:33:50.602894 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:50.602977 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:50.603033 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 04:33:50.603076 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 04:33:50.611982 5456 raft_consensus.cc:1261] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P da8bc6f2729a4c7dbcfb709d2c27e8db [term 1 FOLLOWER]: Refusing update from remote peer 873cc61681de413cbcbadff4ca36c658: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:33:50.612632 5326 raft_consensus.cc:1261] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 5d19408bd2f24b76a6584465d14dbc69 [term 1 FOLLOWER]: Refusing update from remote peer 873cc61681de413cbcbadff4ca36c658: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:33:50.613137 5526 consensus_queue.cc:1035] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da8bc6f2729a4c7dbcfb709d2c27e8db" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:33:50.613483 5530 consensus_queue.cc:1035] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d19408bd2f24b76a6584465d14dbc69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:33:50.624559 5535 mvcc.cc:205] Tried to move back new op lower bound from 7298431510976020480 to 7298431510845206528. Current Snapshot: MvccSnapshot[applied={T|T < 7298431510976020480}]
I20260619 04:33:50.636132 5054 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:33:50.636399 5054 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:33:50.639798 5054 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 276ec85913c14fc2b0ee751e33df0a26: Sending DeleteTablet for 3 replicas of tablet 9ed05ec9d1a64c18b9b8061f536ca3f0
I20260619 04:33:50.640801 5306 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed05ec9d1a64c18b9b8061f536ca3f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:50 UTC) from {username='slave'} at 127.0.0.1:46556
I20260619 04:33:50.640952 5176 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed05ec9d1a64c18b9b8061f536ca3f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:50 UTC) from {username='slave'} at 127.0.0.1:45684
I20260619 04:33:50.641827 5543 tablet_replica.cc:330] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658: stopping tablet replica
I20260619 04:33:50.641783 5436 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed05ec9d1a64c18b9b8061f536ca3f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:33:50 UTC) from {username='slave'} at 127.0.0.1:32938
I20260619 04:33:50.642338 5543 raft_consensus.cc:2229] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:33:50.642977 5543 raft_consensus.cc:2258] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:50.643633 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 5114
I20260619 04:33:50.645799 5543 ts_tablet_manager.cc:1905] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:50.653163 5543 ts_tablet_manager.cc:1918] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:33:50.653409 5543 log.cc:1196] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-0/wal/wals/9ed05ec9d1a64c18b9b8061f536ca3f0
I20260619 04:33:50.654021 5543 ts_tablet_manager.cc:1939] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 873cc61681de413cbcbadff4ca36c658: Deleting consensus metadata
I20260619 04:33:50.655798 5041 catalog_manager.cc:4841] TS 873cc61681de413cbcbadff4ca36c658 (127.0.97.1:43487): tablet 9ed05ec9d1a64c18b9b8061f536ca3f0 (table test-table [id=b766d352706a435b9346d2973e82a3c1]) successfully deleted
I20260619 04:33:50.656874 5544 tablet_replica.cc:330] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P da8bc6f2729a4c7dbcfb709d2c27e8db: stopping tablet replica
I20260619 04:33:50.657281 5544 raft_consensus.cc:2229] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P da8bc6f2729a4c7dbcfb709d2c27e8db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:33:50.657752 5544 raft_consensus.cc:2258] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P da8bc6f2729a4c7dbcfb709d2c27e8db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:50.659193 5542 tablet_replica.cc:330] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 5d19408bd2f24b76a6584465d14dbc69: stopping tablet replica
I20260619 04:33:50.659566 5542 raft_consensus.cc:2229] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 5d19408bd2f24b76a6584465d14dbc69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:33:50.660017 5542 raft_consensus.cc:2258] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 5d19408bd2f24b76a6584465d14dbc69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:33:50.660027 5544 ts_tablet_manager.cc:1905] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P da8bc6f2729a4c7dbcfb709d2c27e8db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:50.661177 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:50.644834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45698 (local address 127.0.97.1:43487)
0619 04:33:50.644995 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:50.645001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:50.658524 (+ 13523us) server_negotiation.cc:407] Connection header received
0619 04:33:50.658757 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:50.658762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:50.658826 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:50.658932 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:50.660004 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:50.660010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:50.660015 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:33:50.660034 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:50.660067 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:33:50.660081 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:50.660168 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:50.660861 (+ 693us) server_negotiation.cc:299] Negotiation successful
0619 04:33:50.660916 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 04:33:50.661855 5542 ts_tablet_manager.cc:1905] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 5d19408bd2f24b76a6584465d14dbc69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:33:50.661963 5236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:50.667342 5544 ts_tablet_manager.cc:1918] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P da8bc6f2729a4c7dbcfb709d2c27e8db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:33:50.667486 5544 log.cc:1196] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P da8bc6f2729a4c7dbcfb709d2c27e8db: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/9ed05ec9d1a64c18b9b8061f536ca3f0
I20260619 04:33:50.667866 5544 ts_tablet_manager.cc:1939] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P da8bc6f2729a4c7dbcfb709d2c27e8db: Deleting consensus metadata
I20260619 04:33:50.668439 5542 ts_tablet_manager.cc:1918] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 5d19408bd2f24b76a6584465d14dbc69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:33:50.668622 5542 log.cc:1196] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 5d19408bd2f24b76a6584465d14dbc69: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/9ed05ec9d1a64c18b9b8061f536ca3f0
I20260619 04:33:50.669064 5542 ts_tablet_manager.cc:1939] T 9ed05ec9d1a64c18b9b8061f536ca3f0 P 5d19408bd2f24b76a6584465d14dbc69: Deleting consensus metadata
I20260619 04:33:50.669370 5041 catalog_manager.cc:4841] TS da8bc6f2729a4c7dbcfb709d2c27e8db (127.0.97.3:45341): tablet 9ed05ec9d1a64c18b9b8061f536ca3f0 (table test-table [id=b766d352706a435b9346d2973e82a3c1]) successfully deleted
I20260619 04:33:50.670588 5039 catalog_manager.cc:4841] TS 5d19408bd2f24b76a6584465d14dbc69 (127.0.97.2:41555): tablet 9ed05ec9d1a64c18b9b8061f536ca3f0 (table test-table [id=b766d352706a435b9346d2973e82a3c1]) successfully deleted
I20260619 04:33:50.753854 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 5114
I20260619 04:33:50.767933 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 5244
I20260619 04:33:50.770596 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:50.768669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46572 (local address 127.0.97.2:41555)
0619 04:33:50.768856 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:50.768862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:50.768881 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:33:50.769002 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:50.769006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:50.769091 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:50.769177 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:50.770016 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:50.770022 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:50.770027 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:33:50.770045 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:50.770073 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:33:50.770085 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:50.770196 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:50.770273 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 04:33:50.770330 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 04:33:50.771196 5366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:50.854321 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 5244
I20260619 04:33:50.866917 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 5374
I20260619 04:33:50.869282 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:50.867571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.0.97.3:45341)
0619 04:33:50.867716 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:50.867720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:50.867756 (+ 36us) server_negotiation.cc:407] Connection header received
0619 04:33:50.867904 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:50.867907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:50.867980 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:50.868088 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:50.868826 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:50.868829 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:50.868832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:33:50.868843 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:50.868866 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:33:50.868876 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:50.868970 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:50.869005 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 04:33:50.869068 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 04:33:50.869837 5496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:50.954164 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 5374
I20260619 04:33:50.966240 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 5024
I20260619 04:33:50.967288 5085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:51.067804 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 5024
2026-06-19T04:33:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2525 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 04:33:51.103116 388 mini_oidc.cc:152] Starting JWKS server
I20260619 04:33:51.105597 388 webserver.cc:460] Webserver started at https://127.0.0.1:32777/ using document root <none> and password file <none>
I20260619 04:33:51.105859 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 04:33:51.106437 388 webserver.cc:460] Webserver started at http://127.0.0.1:36781/ using document root <none> and password file <none>
2026-06-19T04:33:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:33:51Z Disabled control of system clock
I20260619 04:33:51.129257 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41155
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41155
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32777/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781843588881932-388-0/testchainca.cert with env {}
W20260619 04:33:51.428051 5555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:51.428313 5555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:51.428395 5555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:51.435293 5555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:33:51.435390 5555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:51.435425 5555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:51.435480 5555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:33:51.435521 5555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 04:33:51.435577 5555 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 04:33:51.435621 5555 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32777/jwks.json
I20260619 04:33:51.445113 5555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41155
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32777/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781843588881932-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:51.446242 5555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:51.447387 5555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:51.457851 5561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:51.457911 5560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:51.458293 5563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:51.458707 5555 server_base.cc:1034] running on GCE node
I20260619 04:33:51.459360 5555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:51.460594 5555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:51.461786 5555 hybrid_clock.cc:648] HybridClock initialized: now 1781843631461736 us; error 41 us; skew 500 ppm
I20260619 04:33:51.462117 5555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:33:51.464141 5555 webserver.cc:460] Webserver started at http://127.0.97.62:34703/ using document root <none> and password file <none>
I20260619 04:33:51.464792 5555 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:51.464893 5555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:51.465159 5555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:51.467773 5555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "6d20022d01c74a9fb4530cab5aee6b2d"
format_stamp: "Formatted at 2026-06-19 04:33:51 on dist-test-slave-n3tq"
server_key: "a5eeeb6e3f0f09ee6b1f1543d723b417"
server_key_iv: "d58f17eac215909c489ff358aabdb693"
server_key_version: "encryptionkey@0"
I20260619 04:33:51.468513 5555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "6d20022d01c74a9fb4530cab5aee6b2d"
format_stamp: "Formatted at 2026-06-19 04:33:51 on dist-test-slave-n3tq"
server_key: "a5eeeb6e3f0f09ee6b1f1543d723b417"
server_key_iv: "d58f17eac215909c489ff358aabdb693"
server_key_version: "encryptionkey@0"
I20260619 04:33:51.473402 5555 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 04:33:51.476439 5569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:51.477697 5555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260619 04:33:51.477846 5555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "6d20022d01c74a9fb4530cab5aee6b2d"
format_stamp: "Formatted at 2026-06-19 04:33:51 on dist-test-slave-n3tq"
server_key: "a5eeeb6e3f0f09ee6b1f1543d723b417"
server_key_iv: "d58f17eac215909c489ff358aabdb693"
server_key_version: "encryptionkey@0"
I20260619 04:33:51.478024 5555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:51.504279 5555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:51.505117 5555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:51.505363 5555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:51.520839 5555 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41155
I20260619 04:33:51.520856 5620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41155 every 8 connection(s)
I20260619 04:33:51.522071 5555 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:33:51.524317 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 5555
I20260619 04:33:51.524521 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:33:51.525259 388 external_mini_cluster.cc:1428] Setting key 8fc4c144152523c441353f69fd099e3d
I20260619 04:33:51.526165 5621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:51.537432 5621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d: Bootstrap starting.
I20260619 04:33:51.537475 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:51.527014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49404 (local address 127.0.97.62:41155)
0619 04:33:51.527895 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:51.527905 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:51.527944 (+ 39us) server_negotiation.cc:407] Connection header received
0619 04:33:51.529443 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:51.529463 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:51.529922 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:51.530275 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:51.530843 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:51.532174 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:51.533496 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:51.534113 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:51.534294 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:51.535362 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:51.535370 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:51.535385 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:33:51.535904 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:51.535970 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:33:51.535989 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:51.536107 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:51.536291 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 04:33:51.536600 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":300,"threads_started":1}
I20260619 04:33:51.540699 5621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:51.540995 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:33:51.541631 5621 log.cc:824] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:51.544158 5621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d: No bootstrap required, opened a new log
I20260619 04:33:51.551043 5621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d20022d01c74a9fb4530cab5aee6b2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41155 } }
I20260619 04:33:51.551363 5621 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:51.551450 5621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d20022d01c74a9fb4530cab5aee6b2d, State: Initialized, Role: FOLLOWER
I20260619 04:33:51.551973 5621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d20022d01c74a9fb4530cab5aee6b2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41155 } }
I20260619 04:33:51.552179 5621 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:33:51.552294 5621 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:33:51.552440 5621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:51.554422 5621 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d20022d01c74a9fb4530cab5aee6b2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41155 } }
I20260619 04:33:51.554868 5621 leader_election.cc:302] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d20022d01c74a9fb4530cab5aee6b2d; no voters:
I20260619 04:33:51.555429 5621 leader_election.cc:288] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:33:51.555575 5626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:51.556346 5626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d [term 1 LEADER]: Becoming Leader. State: Replica: 6d20022d01c74a9fb4530cab5aee6b2d, State: Running, Role: LEADER
I20260619 04:33:51.556845 5626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d20022d01c74a9fb4530cab5aee6b2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41155 } }
I20260619 04:33:51.557399 5621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:33:51.559337 5627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d20022d01c74a9fb4530cab5aee6b2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d20022d01c74a9fb4530cab5aee6b2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41155 } } }
I20260619 04:33:51.559725 5627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:51.560521 5628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d20022d01c74a9fb4530cab5aee6b2d. Latest consensus state: current_term: 1 leader_uuid: "6d20022d01c74a9fb4530cab5aee6b2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d20022d01c74a9fb4530cab5aee6b2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41155 } } }
I20260619 04:33:51.560818 5628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:51.563163 5635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:33:51.567912 5635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:33:51.575754 5635 catalog_manager.cc:1347] Generated new cluster ID: ae662e7e6d2b4d7e952efd3937a07a3b
I20260619 04:33:51.575851 5635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:33:51.586115 5635 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:33:51.587561 5635 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:33:51.598208 5635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d20022d01c74a9fb4530cab5aee6b2d: Generated new TSK 0
I20260619 04:33:51.598937 5635 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 04:33:51.644901 5651 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 04:33:51.645074 5651 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 04:33:51.645824 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:51.617583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49408 (local address 127.0.97.62:41155)
0619 04:33:51.617784 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:51.617791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:51.618224 (+ 433us) server_negotiation.cc:407] Connection header received
0619 04:33:51.618558 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:51.618562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:51.618633 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:51.618750 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0619 04:33:51.619372 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:51.620107 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:51.622147 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:51.622570 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:51.622666 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:51.623268 (+ 602us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 04:33:51.645299 (+ 22031us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32777/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 04:33:51.645599 (+ 300us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49408: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32777/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 04:33:51.645936 5624 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49408: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32777/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 04:33:51.647658 5650 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:33:51.617474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41155 (local address 127.0.0.1:49408)
0619 04:33:51.618105 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:51.618123 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:51.618323 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:51.618810 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:51.618815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:51.618829 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0619 04:33:51.619206 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:51.619218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:51.620253 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:51.620259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:51.621995 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:51.622010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:51.622679 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:51.622749 (+ 70us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:51.622933 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:51.623011 (+ 78us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 04:33:51.645589 (+ 22578us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32777/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 04:33:51.645939 (+ 350us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:41155: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32777/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":309,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:41155: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32777/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 04:33:51.649645 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 5555
I20260619 04:33:51.650799 5616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:51.740756 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 5555
2026-06-19T04:33:51Z chronyd exiting
I20260619 04:33:51.787016 388 test_util.cc:175] -----------------------------------------------
I20260619 04:33:51.787137 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781843588881932-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (688 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 04:33:51.790833 388 mini_oidc.cc:152] Starting JWKS server
I20260619 04:33:51.792517 388 webserver.cc:460] Webserver started at https://127.0.0.1:36485/ using document root <none> and password file <none>
I20260619 04:33:51.792732 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 04:33:51.793325 388 webserver.cc:460] Webserver started at http://127.0.0.1:41005/ using document root <none> and password file <none>
2026-06-19T04:33:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:33:51Z Disabled control of system clock
I20260619 04:33:51.815662 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33719
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33719
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36485/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781843588881932-388-0/testchainca.cert with env {}
W20260619 04:33:52.114943 5658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:52.115242 5658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:52.115352 5658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:52.122347 5658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:33:52.122465 5658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:52.122526 5658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:52.122601 5658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:33:52.122663 5658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 04:33:52.122746 5658 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 04:33:52.122798 5658 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36485/jwks.json
I20260619 04:33:52.132283 5658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33719
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36485/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781843588881932-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:52.133512 5658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:52.134742 5658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:52.145467 5663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:52.145473 5666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:52.145494 5664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:52.146510 5658 server_base.cc:1034] running on GCE node
I20260619 04:33:52.147158 5658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:52.148388 5658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:52.149593 5658 hybrid_clock.cc:648] HybridClock initialized: now 1781843632149555 us; error 29 us; skew 500 ppm
I20260619 04:33:52.149895 5658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:33:52.151829 5658 webserver.cc:460] Webserver started at http://127.0.97.62:34763/ using document root <none> and password file <none>
I20260619 04:33:52.152432 5658 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:52.152567 5658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:52.152848 5658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:52.155411 5658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "d3d78bc9e3c649b293fb318685c7cc6d"
format_stamp: "Formatted at 2026-06-19 04:33:52 on dist-test-slave-n3tq"
server_key: "4bf58d59bb1284522ba0815c4ecd38b7"
server_key_iv: "8af62a476b136ae760185338f8e27408"
server_key_version: "encryptionkey@0"
I20260619 04:33:52.156138 5658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "d3d78bc9e3c649b293fb318685c7cc6d"
format_stamp: "Formatted at 2026-06-19 04:33:52 on dist-test-slave-n3tq"
server_key: "4bf58d59bb1284522ba0815c4ecd38b7"
server_key_iv: "8af62a476b136ae760185338f8e27408"
server_key_version: "encryptionkey@0"
I20260619 04:33:52.160604 5658 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260619 04:33:52.163612 5672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:52.164768 5658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 04:33:52.164906 5658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "d3d78bc9e3c649b293fb318685c7cc6d"
format_stamp: "Formatted at 2026-06-19 04:33:52 on dist-test-slave-n3tq"
server_key: "4bf58d59bb1284522ba0815c4ecd38b7"
server_key_iv: "8af62a476b136ae760185338f8e27408"
server_key_version: "encryptionkey@0"
I20260619 04:33:52.165074 5658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:52.200851 5658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:52.201653 5658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:52.201905 5658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:52.217926 5658 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33719
I20260619 04:33:52.217945 5723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33719 every 8 connection(s)
I20260619 04:33:52.219173 5658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:33:52.222786 5724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:52.229501 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 5658
I20260619 04:33:52.229693 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:33:52.230253 388 external_mini_cluster.cc:1428] Setting key 61dfa7739138ae78018aab7664e7129d
I20260619 04:33:52.232779 5724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d: Bootstrap starting.
I20260619 04:33:52.235807 5724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:52.236922 5724 log.cc:824] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:52.239837 5724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d: No bootstrap required, opened a new log
I20260619 04:33:52.243543 5727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:52.232017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60918 (local address 127.0.97.62:33719)
0619 04:33:52.233036 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:52.233048 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:52.233088 (+ 40us) server_negotiation.cc:407] Connection header received
0619 04:33:52.234740 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:52.234762 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:52.235218 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:52.235572 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:52.236128 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:52.237528 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:52.239154 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:52.239822 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:52.239978 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:52.241134 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:52.241142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:52.241155 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:33:52.241704 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:52.241783 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:33:52.241806 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:52.241897 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:52.242112 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 04:33:52.242442 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":355,"threads_started":1}
I20260619 04:33:52.247203 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:33:52.248442 5724 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d78bc9e3c649b293fb318685c7cc6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33719 } }
I20260619 04:33:52.248880 5724 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:52.248968 5724 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d78bc9e3c649b293fb318685c7cc6d, State: Initialized, Role: FOLLOWER
I20260619 04:33:52.249568 5724 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d78bc9e3c649b293fb318685c7cc6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33719 } }
I20260619 04:33:52.249971 5724 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:33:52.250123 5724 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:33:52.250342 5724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:52.252534 5724 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d78bc9e3c649b293fb318685c7cc6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33719 } }
I20260619 04:33:52.253010 5724 leader_election.cc:302] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3d78bc9e3c649b293fb318685c7cc6d; no voters:
I20260619 04:33:52.253575 5724 leader_election.cc:288] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:33:52.253706 5729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:52.255273 5724 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:33:52.255594 5729 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d [term 1 LEADER]: Becoming Leader. State: Replica: d3d78bc9e3c649b293fb318685c7cc6d, State: Running, Role: LEADER
I20260619 04:33:52.256101 5729 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d78bc9e3c649b293fb318685c7cc6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33719 } }
I20260619 04:33:52.259058 5730 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3d78bc9e3c649b293fb318685c7cc6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d78bc9e3c649b293fb318685c7cc6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33719 } } }
I20260619 04:33:52.259021 5731 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3d78bc9e3c649b293fb318685c7cc6d. Latest consensus state: current_term: 1 leader_uuid: "d3d78bc9e3c649b293fb318685c7cc6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d78bc9e3c649b293fb318685c7cc6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33719 } } }
I20260619 04:33:52.259466 5730 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:52.259472 5731 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:52.264612 5739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:33:52.269356 5739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:33:52.278489 5739 catalog_manager.cc:1347] Generated new cluster ID: 4b85666b126a4b43b3b5d2db1a040192
I20260619 04:33:52.278604 5739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:33:52.295979 5739 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:33:52.297466 5739 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:33:52.308275 5739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3d78bc9e3c649b293fb318685c7cc6d: Generated new TSK 0
I20260619 04:33:52.309324 5739 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 04:33:52.344148 5754 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 04:33:52.344276 5754 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 04:33:52.345269 5727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:52.325572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60924 (local address 127.0.97.62:33719)
0619 04:33:52.325775 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:52.325781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:52.326140 (+ 359us) server_negotiation.cc:407] Connection header received
0619 04:33:52.326351 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:52.326355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:52.326420 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:52.326554 (+ 134us) server_negotiation.cc:226] Negotiated authn=JWT
0619 04:33:52.327064 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:52.327762 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:52.329795 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:52.330309 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:52.330429 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:52.331045 (+ 616us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 04:33:52.344773 (+ 13728us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36485/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 04:33:52.345066 (+ 293us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60924: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36485/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 04:33:52.345391 5727 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60924: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36485/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 04:33:52.345711 5753 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:33:52.325398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33719 (local address 127.0.0.1:60924)
0619 04:33:52.326039 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:52.326060 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:52.326204 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:52.326586 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:52.326591 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:52.326604 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0619 04:33:52.326907 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:52.326919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:52.327915 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:52.327921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:52.329624 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:52.329640 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:52.330435 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:52.330481 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:52.330692 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:52.330744 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 04:33:52.345033 (+ 14289us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36485/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 04:33:52.345321 (+ 288us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:33719: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36485/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":322,"threads_started":1}
I20260619 04:33:52.347334 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 5658
I20260619 04:33:52.348467 5719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:52.437291 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 5658
2026-06-19T04:33:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (698 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 04:33:52.489243 388 mini_oidc.cc:152] Starting JWKS server
I20260619 04:33:52.491035 388 webserver.cc:460] Webserver started at https://127.0.0.1:45961/ using document root <none> and password file <none>
I20260619 04:33:52.491219 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 04:33:52.491741 388 webserver.cc:460] Webserver started at http://127.0.0.1:45651/ using document root <none> and password file <none>
2026-06-19T04:33:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:33:52Z Disabled control of system clock
I20260619 04:33:52.514149 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45297
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45297
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45961/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 04:33:52.810973 5761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:52.811270 5761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:52.811385 5761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:52.818293 5761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:33:52.818400 5761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:52.818467 5761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:52.818544 5761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:33:52.818603 5761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 04:33:52.818684 5761 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 04:33:52.818735 5761 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45961/jwks.json
I20260619 04:33:52.827406 5761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45297
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45961/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:52.828651 5761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:52.830327 5761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:52.840672 5766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:52.840795 5767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:52.840839 5769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:52.841791 5761 server_base.cc:1034] running on GCE node
I20260619 04:33:52.842473 5761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:52.843775 5761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:52.844956 5761 hybrid_clock.cc:648] HybridClock initialized: now 1781843632844907 us; error 45 us; skew 500 ppm
I20260619 04:33:52.845268 5761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:33:52.850248 5761 webserver.cc:460] Webserver started at http://127.0.97.62:41845/ using document root <none> and password file <none>
I20260619 04:33:52.850884 5761 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:52.850977 5761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:52.851276 5761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:52.854005 5761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "62039a09d7494fcc81f82177b48f3a6c"
format_stamp: "Formatted at 2026-06-19 04:33:52 on dist-test-slave-n3tq"
server_key: "1735ca6a3a0b3f472a09df4bfcc4b090"
server_key_iv: "0506e28647d68019022764d7669e639d"
server_key_version: "encryptionkey@0"
I20260619 04:33:52.854727 5761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "62039a09d7494fcc81f82177b48f3a6c"
format_stamp: "Formatted at 2026-06-19 04:33:52 on dist-test-slave-n3tq"
server_key: "1735ca6a3a0b3f472a09df4bfcc4b090"
server_key_iv: "0506e28647d68019022764d7669e639d"
server_key_version: "encryptionkey@0"
I20260619 04:33:52.859474 5761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 04:33:52.862721 5775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:52.864022 5761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 04:33:52.864178 5761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "62039a09d7494fcc81f82177b48f3a6c"
format_stamp: "Formatted at 2026-06-19 04:33:52 on dist-test-slave-n3tq"
server_key: "1735ca6a3a0b3f472a09df4bfcc4b090"
server_key_iv: "0506e28647d68019022764d7669e639d"
server_key_version: "encryptionkey@0"
I20260619 04:33:52.864367 5761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:52.892410 5761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:52.893277 5761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:52.893510 5761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:52.909556 5761 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45297
I20260619 04:33:52.909577 5826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45297 every 8 connection(s)
I20260619 04:33:52.910832 5761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:33:52.914306 5827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:52.918356 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 5761
I20260619 04:33:52.918531 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:33:52.919258 388 external_mini_cluster.cc:1428] Setting key 3d1fe0401021156d0023f561d6ee9aba
I20260619 04:33:52.925400 5827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c: Bootstrap starting.
I20260619 04:33:52.928320 5827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:52.929342 5827 log.cc:824] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:52.932176 5827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c: No bootstrap required, opened a new log
I20260619 04:33:52.932574 5830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:52.921231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48620 (local address 127.0.97.62:45297)
0619 04:33:52.922220 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:52.922231 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:52.922271 (+ 40us) server_negotiation.cc:407] Connection header received
0619 04:33:52.924237 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:52.924260 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:52.924781 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:52.925154 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:52.925698 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:52.926964 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:52.928219 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:52.928957 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:52.929132 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:52.930241 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:52.930252 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:52.930269 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:33:52.930872 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:52.930946 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:33:52.930966 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:52.931053 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:52.931266 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 04:33:52.931560 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":340,"threads_started":1}
I20260619 04:33:52.936302 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:33:52.941288 5827 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62039a09d7494fcc81f82177b48f3a6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45297 } }
I20260619 04:33:52.941646 5827 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:52.941746 5827 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62039a09d7494fcc81f82177b48f3a6c, State: Initialized, Role: FOLLOWER
I20260619 04:33:52.942309 5827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62039a09d7494fcc81f82177b48f3a6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45297 } }
I20260619 04:33:52.942504 5827 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:33:52.942622 5827 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:33:52.942775 5827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:52.945050 5827 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62039a09d7494fcc81f82177b48f3a6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45297 } }
I20260619 04:33:52.945536 5827 leader_election.cc:302] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62039a09d7494fcc81f82177b48f3a6c; no voters:
I20260619 04:33:52.946079 5827 leader_election.cc:288] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:33:52.946221 5832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:52.946866 5832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c [term 1 LEADER]: Becoming Leader. State: Replica: 62039a09d7494fcc81f82177b48f3a6c, State: Running, Role: LEADER
I20260619 04:33:52.947402 5832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62039a09d7494fcc81f82177b48f3a6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45297 } }
I20260619 04:33:52.947983 5827 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:33:52.949975 5833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62039a09d7494fcc81f82177b48f3a6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62039a09d7494fcc81f82177b48f3a6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45297 } } }
I20260619 04:33:52.949960 5834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62039a09d7494fcc81f82177b48f3a6c. Latest consensus state: current_term: 1 leader_uuid: "62039a09d7494fcc81f82177b48f3a6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62039a09d7494fcc81f82177b48f3a6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45297 } } }
I20260619 04:33:52.950371 5833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:52.950541 5834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:52.953531 5839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:33:52.958429 5839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:33:52.967077 5839 catalog_manager.cc:1347] Generated new cluster ID: 816fa508edb64c2fb5be8109751974ad
I20260619 04:33:52.967185 5839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:33:52.983527 5839 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:33:52.984742 5839 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:33:52.999678 5839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62039a09d7494fcc81f82177b48f3a6c: Generated new TSK 0
I20260619 04:33:53.000744 5839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:33:53.027971 5830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:53.020104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48634 (local address 127.0.97.62:45297)
0619 04:33:53.020279 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:53.020286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:53.020619 (+ 333us) server_negotiation.cc:407] Connection header received
0619 04:33:53.020817 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:53.020823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:53.020912 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:53.021074 (+ 162us) server_negotiation.cc:226] Negotiated authn=JWT
0619 04:33:53.021493 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:53.022569 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:53.023891 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:53.024576 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:53.024695 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:53.025123 (+ 428us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 04:33:53.027357 (+ 2234us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45961/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 04:33:53.027658 (+ 301us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48634: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45961/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 04:33:53.028102 5830 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48634: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45961/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 04:33:53.029604 5856 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:33:53.019942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45297 (local address 127.0.0.1:48634)
0619 04:33:53.020530 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:53.020546 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:53.020666 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:53.021095 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:53.021098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:53.021108 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0619 04:33:53.021364 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:53.021372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:53.022710 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:53.022713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:53.023764 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:53.023775 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:53.024691 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:53.024747 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:53.024861 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:53.024892 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 04:33:53.027549 (+ 2657us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45961/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 04:33:53.029219 (+ 1670us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:45297: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45961/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":1390,"thread_start_us":288,"threads_started":1}
I20260619 04:33:53.030926 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 5761
I20260619 04:33:53.032014 5822 generic_service.cc:196] Checking for leaks (request via RPC)
E20260619 04:33:53.042409 5857 webserver.cc:533] Webserver: sslize failed:
E20260619 04:33:53.042539 5857 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 04:33:53.124267 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 5761
2026-06-19T04:33:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (685 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T04:33:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:33:53Z Disabled control of system clock
I20260619 04:33:53.197954 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41123
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41123
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781843588881932-388-0/testchainca.cert with env {}
W20260619 04:33:53.499558 5862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:53.499868 5862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:53.500000 5862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:53.507155 5862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:33:53.507282 5862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:53.507347 5862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:53.507426 5862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:33:53.507484 5862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 04:33:53.507567 5862 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 04:33:53.507620 5862 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 04:33:53.517495 5862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41123
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781843588881932-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:53.518746 5862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:53.519954 5862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:53.530893 5870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:53.530934 5868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:53.530905 5867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:53.531863 5862 server_base.cc:1034] running on GCE node
I20260619 04:33:53.532611 5862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:53.533953 5862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:53.535151 5862 hybrid_clock.cc:648] HybridClock initialized: now 1781843633535107 us; error 33 us; skew 500 ppm
I20260619 04:33:53.535471 5862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:33:53.537617 5862 webserver.cc:460] Webserver started at http://127.0.97.62:35777/ using document root <none> and password file <none>
I20260619 04:33:53.538269 5862 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:53.538367 5862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:53.538645 5862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:53.541410 5862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "0fff9b7bf3ed4785890d37bf9f7d1168"
format_stamp: "Formatted at 2026-06-19 04:33:53 on dist-test-slave-n3tq"
server_key: "6790ce3eca0a26d7b8732a55e127b55d"
server_key_iv: "81d8c5a5efe72728ed659ea297df4e0a"
server_key_version: "encryptionkey@0"
I20260619 04:33:53.542146 5862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "0fff9b7bf3ed4785890d37bf9f7d1168"
format_stamp: "Formatted at 2026-06-19 04:33:53 on dist-test-slave-n3tq"
server_key: "6790ce3eca0a26d7b8732a55e127b55d"
server_key_iv: "81d8c5a5efe72728ed659ea297df4e0a"
server_key_version: "encryptionkey@0"
I20260619 04:33:53.546805 5862 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 04:33:53.549988 5876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:53.551244 5862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:33:53.551393 5862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "0fff9b7bf3ed4785890d37bf9f7d1168"
format_stamp: "Formatted at 2026-06-19 04:33:53 on dist-test-slave-n3tq"
server_key: "6790ce3eca0a26d7b8732a55e127b55d"
server_key_iv: "81d8c5a5efe72728ed659ea297df4e0a"
server_key_version: "encryptionkey@0"
I20260619 04:33:53.551570 5862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:53.569892 5862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:53.570714 5862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:53.570994 5862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:53.587244 5862 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41123
I20260619 04:33:53.587268 5927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41123 every 8 connection(s)
I20260619 04:33:53.588513 5862 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:33:53.592142 5928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:53.593437 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 5862
I20260619 04:33:53.593622 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:33:53.594237 388 external_mini_cluster.cc:1428] Setting key 4dbae414e0200cfd9259007fcb0d9f77
I20260619 04:33:53.605384 5928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168: Bootstrap starting.
I20260619 04:33:53.607095 5931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:53.596012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53800 (local address 127.0.97.62:41123)
0619 04:33:53.596996 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:53.597006 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:53.597055 (+ 49us) server_negotiation.cc:407] Connection header received
0619 04:33:53.598537 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:53.598559 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:53.599030 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:53.599468 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:53.599992 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:53.601270 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:53.603010 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:53.603668 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:53.603786 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:53.604964 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:53.604972 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:53.604986 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:33:53.605525 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:53.605605 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:33:53.605624 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:53.605704 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:53.605898 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 04:33:53.606187 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":346,"threads_started":1}
I20260619 04:33:53.608589 5928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168: Neither blocks nor log segments found. Creating new log.
I20260619 04:33:53.609742 5928 log.cc:824] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:53.610707 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:33:53.612360 5928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168: No bootstrap required, opened a new log
I20260619 04:33:53.619376 5928 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff9b7bf3ed4785890d37bf9f7d1168" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41123 } }
I20260619 04:33:53.619731 5928 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:53.619822 5928 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fff9b7bf3ed4785890d37bf9f7d1168, State: Initialized, Role: FOLLOWER
I20260619 04:33:53.620317 5928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff9b7bf3ed4785890d37bf9f7d1168" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41123 } }
I20260619 04:33:53.620532 5928 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:33:53.620639 5928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:33:53.620790 5928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:53.622745 5928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff9b7bf3ed4785890d37bf9f7d1168" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41123 } }
I20260619 04:33:53.623203 5928 leader_election.cc:302] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fff9b7bf3ed4785890d37bf9f7d1168; no voters:
I20260619 04:33:53.623725 5928 leader_election.cc:288] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:33:53.623847 5933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:53.624505 5933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168 [term 1 LEADER]: Becoming Leader. State: Replica: 0fff9b7bf3ed4785890d37bf9f7d1168, State: Running, Role: LEADER
I20260619 04:33:53.624967 5933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff9b7bf3ed4785890d37bf9f7d1168" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41123 } }
I20260619 04:33:53.625602 5928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:33:53.627449 5935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fff9b7bf3ed4785890d37bf9f7d1168. Latest consensus state: current_term: 1 leader_uuid: "0fff9b7bf3ed4785890d37bf9f7d1168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff9b7bf3ed4785890d37bf9f7d1168" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41123 } } }
I20260619 04:33:53.627810 5935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168 [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:53.629220 5934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fff9b7bf3ed4785890d37bf9f7d1168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff9b7bf3ed4785890d37bf9f7d1168" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41123 } } }
I20260619 04:33:53.629599 5934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168 [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:53.630728 5939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:33:53.635452 5939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:33:53.643865 5939 catalog_manager.cc:1347] Generated new cluster ID: 4231a53b405246be9b512f5ce1276e13
I20260619 04:33:53.643961 5939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:33:53.668570 5939 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:33:53.669770 5939 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:33:53.678963 5939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fff9b7bf3ed4785890d37bf9f7d1168: Generated new TSK 0
I20260619 04:33:53.679718 5939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:33:54.157935 5931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:53.707966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.0.97.62:41123)
0619 04:33:53.708167 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:53.708173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:53.708486 (+ 313us) server_negotiation.cc:407] Connection header received
0619 04:33:53.708712 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:53.708717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:53.708816 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:53.708989 (+ 173us) server_negotiation.cc:226] Negotiated authn=JWT
0619 04:33:53.709358 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:53.710495 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:53.711796 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:53.712505 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:53.712647 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:53.713109 (+ 462us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 04:33:54.157422 (+444313us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 04:33:54.157730 (+ 308us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53806: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 04:33:54.158030 5931 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53806: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 04:33:54.158304 5957 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:33:53.707815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41123 (local address 127.0.0.1:53806)
0619 04:33:53.708372 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:53.708388 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:53.708537 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:53.708989 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:53.708992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:53.709000 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 04:33:53.709230 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:53.709238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:53.710632 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:53.710635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:53.711673 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:53.711683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:53.712649 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:53.712699 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:53.712821 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:53.712854 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 04:33:54.157721 (+444867us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 04:33:54.157957 (+ 236us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:41123: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":284,"threads_started":1}
I20260619 04:33:54.159428 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 5862
I20260619 04:33:54.160416 5923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:54.258633 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 5862
2026-06-19T04:33:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1115 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (385 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (384 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:33:55 dist-test-slave-n3tq krb5kdc[5972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:33:55 dist-test-slave-n3tq krb5kdc[5972](info): set up 2 sockets
Jun 19 04:33:55 dist-test-slave-n3tq krb5kdc[5972](info): commencing operation
krb5kdc: starting...
W20260619 04:33:56.024782 388 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:33:56 dist-test-slave-n3tq krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:33:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:33:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:33:56.200747 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43473
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:56.502161 5985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:56.502449 5985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:56.502570 5985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:56.509820 5985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:33:56.509943 5985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:56.510020 5985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:56.510097 5985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:33:56.510166 5985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:33:56.519743 5985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:56.521056 5985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:56.522363 5985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:56.533205 5990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:56.533159 5993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:56.533126 5991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:56.534073 5985 server_base.cc:1034] running on GCE node
I20260619 04:33:56.534793 5985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:56.536087 5985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:56.537279 5985 hybrid_clock.cc:648] HybridClock initialized: now 1781843636537215 us; error 51 us; skew 500 ppm
I20260619 04:33:56.537583 5985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:56 dist-test-slave-n3tq krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:56.540442 5985 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:33:56.541817 5985 webserver.cc:460] Webserver started at http://127.0.97.62:45453/ using document root <none> and password file <none>
I20260619 04:33:56.542443 5985 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:56.542553 5985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:56.542837 5985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:56.545678 5985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "b7631dfcc51941ddaec1f6955fdd98ae"
format_stamp: "Formatted at 2026-06-19 04:33:56 on dist-test-slave-n3tq"
server_key: "7ed3199c70f4eb227a0f5f989f399099"
server_key_iv: "1080a58cec9ace2783f11d842679e73c"
server_key_version: "encryptionkey@0"
I20260619 04:33:56.546427 5985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "b7631dfcc51941ddaec1f6955fdd98ae"
format_stamp: "Formatted at 2026-06-19 04:33:56 on dist-test-slave-n3tq"
server_key: "7ed3199c70f4eb227a0f5f989f399099"
server_key_iv: "1080a58cec9ace2783f11d842679e73c"
server_key_version: "encryptionkey@0"
I20260619 04:33:56.551332 5985 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 04:33:56.554574 6000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:56.555891 5985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 04:33:56.556044 5985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "b7631dfcc51941ddaec1f6955fdd98ae"
format_stamp: "Formatted at 2026-06-19 04:33:56 on dist-test-slave-n3tq"
server_key: "7ed3199c70f4eb227a0f5f989f399099"
server_key_iv: "1080a58cec9ace2783f11d842679e73c"
server_key_version: "encryptionkey@0"
I20260619 04:33:56.556229 5985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:56.572436 5985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:56.576515 5985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:56.576761 5985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:56.592156 5985 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43473
I20260619 04:33:56.592186 6051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43473 every 8 connection(s)
I20260619 04:33:56.593415 5985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:33:56.596846 6052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:33:56.597168 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 5985
I20260619 04:33:56.597360 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:33:56.598033 388 external_mini_cluster.cc:1428] Setting key 54f933b65adec108502575b2b513bab3
I20260619 04:33:56.610769 6052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae: Bootstrap starting.
I20260619 04:33:56.614476 6052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae: Neither blocks nor log segments found. Creating new log.
Jun 19 04:33:56 dist-test-slave-n3tq krb5kdc[5972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:56.615386 6052 log.cc:824] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae: Log is configured to *not* fsync() on all Append() calls
I20260619 04:33:56.617951 6052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae: No bootstrap required, opened a new log
I20260619 04:33:56.622187 6055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:56.599907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38504 (local address 127.0.97.62:43473)
0619 04:33:56.600640 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:56.600655 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:56.600702 (+ 47us) server_negotiation.cc:407] Connection header received
0619 04:33:56.602743 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:56.602771 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:56.603279 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:56.603689 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:56.605283 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:56.607888 (+ 2605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:56.609666 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:56.610690 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:56.610919 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:56.615563 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:56.615573 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:56.615593 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:56.615631 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:56.618515 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:56.619109 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:56.619116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:56.619121 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:56.619280 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:56.619624 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:56.619627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:56.619629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:56.620132 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:56.620415 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:56.620842 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 04:33:56.621224 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":288,"threads_started":1}
I20260619 04:33:56.625833 6052 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7631dfcc51941ddaec1f6955fdd98ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43473 } }
I20260619 04:33:56.626145 6052 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:33:56.626225 6052 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7631dfcc51941ddaec1f6955fdd98ae, State: Initialized, Role: FOLLOWER
I20260619 04:33:56.626744 6052 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7631dfcc51941ddaec1f6955fdd98ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43473 } }
I20260619 04:33:56.626938 6052 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:33:56.627056 6052 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:33:56.627188 6052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:33:56.629518 6052 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7631dfcc51941ddaec1f6955fdd98ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43473 } }
I20260619 04:33:56.629973 6052 leader_election.cc:302] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7631dfcc51941ddaec1f6955fdd98ae; no voters:
I20260619 04:33:56.630483 6052 leader_election.cc:288] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:33:56.630589 6057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:33:56.631160 6057 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae [term 1 LEADER]: Becoming Leader. State: Replica: b7631dfcc51941ddaec1f6955fdd98ae, State: Running, Role: LEADER
I20260619 04:33:56.631594 6057 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7631dfcc51941ddaec1f6955fdd98ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43473 } }
I20260619 04:33:56.632248 6052 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:33:56.634203 6059 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7631dfcc51941ddaec1f6955fdd98ae. Latest consensus state: current_term: 1 leader_uuid: "b7631dfcc51941ddaec1f6955fdd98ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7631dfcc51941ddaec1f6955fdd98ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43473 } } }
I20260619 04:33:56.634588 6059 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:56.634680 6058 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7631dfcc51941ddaec1f6955fdd98ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7631dfcc51941ddaec1f6955fdd98ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43473 } } }
I20260619 04:33:56.634970 6058 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae [sys.catalog]: This master's current role is: LEADER
I20260619 04:33:56.636777 6065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:33:56.641641 6065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:33:56.650095 6065 catalog_manager.cc:1347] Generated new cluster ID: a05c7927c25f4b39a14b2680af49ceb8
I20260619 04:33:56.650203 6065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:33:56.662047 6065 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:33:56.663589 6065 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:33:56.669685 6065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7631dfcc51941ddaec1f6955fdd98ae: Generated new TSK 0
I20260619 04:33:56.670459 6065 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 04:33:56.713017 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43473
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:57.018811 6078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:57.019079 6078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:57.019207 6078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:57.026229 6078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:57.026331 6078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:57.026481 6078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:33:57.035988 6078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:57.037367 6078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:57.038602 6078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:57.050840 6086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:57.050880 6084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:57.050846 6083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:57.055203 6078 server_base.cc:1034] running on GCE node
I20260619 04:33:57.055835 6078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:57.056821 6078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:57.058023 6078 hybrid_clock.cc:648] HybridClock initialized: now 1781843637057977 us; error 37 us; skew 500 ppm
I20260619 04:33:57.058337 6078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:57 dist-test-slave-n3tq krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:57.061465 6078 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:33:57.062772 6078 webserver.cc:460] Webserver started at http://127.0.97.1:42311/ using document root <none> and password file <none>
I20260619 04:33:57.063422 6078 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:57.063513 6078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:57.063746 6078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:57.066531 6078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "3fa94bab8bda4fffa3d7e55e137953e8"
format_stamp: "Formatted at 2026-06-19 04:33:57 on dist-test-slave-n3tq"
server_key: "6915ecbb2278bb720b3f80abf2d33973"
server_key_iv: "dc54fef1820a2d51a2c7f984deec5633"
server_key_version: "encryptionkey@0"
I20260619 04:33:57.067219 6078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3fa94bab8bda4fffa3d7e55e137953e8"
format_stamp: "Formatted at 2026-06-19 04:33:57 on dist-test-slave-n3tq"
server_key: "6915ecbb2278bb720b3f80abf2d33973"
server_key_iv: "dc54fef1820a2d51a2c7f984deec5633"
server_key_version: "encryptionkey@0"
I20260619 04:33:57.071918 6078 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260619 04:33:57.075124 6093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:57.076370 6078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:33:57.076524 6078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "3fa94bab8bda4fffa3d7e55e137953e8"
format_stamp: "Formatted at 2026-06-19 04:33:57 on dist-test-slave-n3tq"
server_key: "6915ecbb2278bb720b3f80abf2d33973"
server_key_iv: "dc54fef1820a2d51a2c7f984deec5633"
server_key_version: "encryptionkey@0"
I20260619 04:33:57.076699 6078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:57.091113 6078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:57.095003 6078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:57.095259 6078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:57.096138 6078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:57.097388 6078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:57.097455 6078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:57.097568 6078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:57.097613 6078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:57.125555 6078 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42419
I20260619 04:33:57.125591 6205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42419 every 8 connection(s)
I20260619 04:33:57.126720 6078 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:33:57.129808 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 6078
I20260619 04:33:57.130026 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:33:57.130775 388 external_mini_cluster.cc:1428] Setting key 433fc691085291582115aa81d8f91359
Jun 19 04:33:57 dist-test-slave-n3tq krb5kdc[5972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:57.146021 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:57.129029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40151 (local address 127.0.97.62:43473)
0619 04:33:57.129444 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:57.129448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:57.130535 (+ 1087us) server_negotiation.cc:407] Connection header received
0619 04:33:57.132520 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:57.132525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:57.132602 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:57.132696 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:57.134186 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:57.134881 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:57.136401 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:57.136872 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:57.136977 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:57.140614 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:57.140619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:57.140622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:57.140654 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:57.143100 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:57.143870 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:57.143874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:57.143876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:57.143992 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:57.144404 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:57.144407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:57.144409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:57.144679 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:57.144819 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:57.145625 (+ 806us) server_negotiation.cc:299] Negotiation successful
0619 04:33:57.145777 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":200,"threads_started":1}
I20260619 04:33:57.147583 6209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:57.129610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43473 (local address 127.0.97.1:40151)
0619 04:33:57.130387 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:57.130430 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:57.132222 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:57.132932 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:57.132941 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:57.133482 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:57.134038 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:57.134055 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:57.135020 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:57.135024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:57.136267 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:57.136280 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:57.137072 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:57.137079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:57.137297 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:57.137900 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:57.137925 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:57.140385 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:57.143224 (+ 2839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:57.143233 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:57.143249 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:57.143766 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:57.144081 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:57.144086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:57.144090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:57.144319 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:57.144815 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:57.144823 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:57.145267 (+ 444us) client_negotiation.cc:769] Sending connection context
0619 04:33:57.145577 (+ 310us) client_negotiation.cc:240] Negotiation successful
0619 04:33:57.146202 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":313,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 04:33:57.149107 6206 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43473
I20260619 04:33:57.149389 6206 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:57.150061 6206 heartbeater.cc:511] Master 127.0.97.62:43473 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 04:33:57.151964 6017 ts_manager.cc:194] Registered new tserver with Master: 3fa94bab8bda4fffa3d7e55e137953e8 (127.0.97.1:42419)
I20260619 04:33:57.153396 6017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 04:33:57.171064 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43473
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:57.472064 6212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:57.472342 6212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:57.472486 6212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:57.479482 6212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:57.479581 6212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:57.479722 6212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:33:57.489023 6212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:57.490185 6212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:57.491544 6212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:57.503507 6218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:57.503612 6217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:57.503530 6220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:57.504426 6212 server_base.cc:1034] running on GCE node
I20260619 04:33:57.504998 6212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:57.505856 6212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:57.507046 6212 hybrid_clock.cc:648] HybridClock initialized: now 1781843637507000 us; error 36 us; skew 500 ppm
I20260619 04:33:57.507335 6212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:57 dist-test-slave-n3tq krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:57.510308 6212 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:33:57.511653 6212 webserver.cc:460] Webserver started at http://127.0.97.2:34089/ using document root <none> and password file <none>
I20260619 04:33:57.512272 6212 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:57.512390 6212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:57.512682 6212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:57.515426 6212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "7bab5116af634a1a9b57c6c9f42215c2"
format_stamp: "Formatted at 2026-06-19 04:33:57 on dist-test-slave-n3tq"
server_key: "ee56ae9242b2ad6448449aec38cc2a47"
server_key_iv: "9170d0f5d7004f246cc5529225afa986"
server_key_version: "encryptionkey@0"
I20260619 04:33:57.516150 6212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7bab5116af634a1a9b57c6c9f42215c2"
format_stamp: "Formatted at 2026-06-19 04:33:57 on dist-test-slave-n3tq"
server_key: "ee56ae9242b2ad6448449aec38cc2a47"
server_key_iv: "9170d0f5d7004f246cc5529225afa986"
server_key_version: "encryptionkey@0"
I20260619 04:33:57.520852 6212 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260619 04:33:57.523985 6227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:57.525254 6212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:33:57.525399 6212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "7bab5116af634a1a9b57c6c9f42215c2"
format_stamp: "Formatted at 2026-06-19 04:33:57 on dist-test-slave-n3tq"
server_key: "ee56ae9242b2ad6448449aec38cc2a47"
server_key_iv: "9170d0f5d7004f246cc5529225afa986"
server_key_version: "encryptionkey@0"
I20260619 04:33:57.525568 6212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:57.543358 6212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:57.547053 6212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:57.547308 6212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:57.548182 6212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:57.549474 6212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:57.549544 6212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:57.549661 6212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:57.549711 6212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:57.577749 6212 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44289
I20260619 04:33:57.577801 6339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44289 every 8 connection(s)
I20260619 04:33:57.578905 6212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:33:57.586802 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 6212
I20260619 04:33:57.586961 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:33:57.587352 388 external_mini_cluster.cc:1428] Setting key c47c84b86898874e626eb0c612e6006d
Jun 19 04:33:57 dist-test-slave-n3tq krb5kdc[5972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:57.598527 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:57.581324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48243 (local address 127.0.97.62:43473)
0619 04:33:57.581469 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:57.581473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:57.582544 (+ 1071us) server_negotiation.cc:407] Connection header received
0619 04:33:57.584212 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:57.584218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:57.584302 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:57.584384 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:57.585799 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:57.586531 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:57.588143 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:57.588652 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:57.588762 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:57.592228 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:57.592235 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:57.592240 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:57.592277 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:57.595478 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:57.596248 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:57.596253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:57.596257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:57.596397 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:57.596821 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:57.596826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:57.596829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:57.597164 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:57.597310 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:57.597993 (+ 683us) server_negotiation.cc:299] Negotiation successful
0619 04:33:57.598200 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 04:33:57.599588 6342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:57.581744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43473 (local address 127.0.97.2:48243)
0619 04:33:57.582395 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:57.582435 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:57.583963 (+ 1528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:57.584638 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:57.584646 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:57.585126 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:57.585643 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:57.585659 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:57.586703 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:57.586707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:57.588025 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:57.588038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:57.588873 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:57.588880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:57.589151 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:57.589757 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:57.589781 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:57.592030 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:57.595594 (+ 3564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:57.595602 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:57.595617 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:57.596134 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:57.596504 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:57.596509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:57.596512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:57.596713 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:57.597298 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:57.597305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:57.597651 (+ 346us) client_negotiation.cc:769] Sending connection context
0619 04:33:57.597910 (+ 259us) client_negotiation.cc:240] Negotiation successful
0619 04:33:57.598376 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":255,"threads_started":1}
I20260619 04:33:57.600976 6340 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43473
I20260619 04:33:57.601256 6340 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:57.602008 6340 heartbeater.cc:511] Master 127.0.97.62:43473 requested a full tablet report, sending...
I20260619 04:33:57.603529 6017 ts_manager.cc:194] Registered new tserver with Master: 7bab5116af634a1a9b57c6c9f42215c2 (127.0.97.2:44289)
I20260619 04:33:57.604671 6017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48243
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 04:33:57.627135 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43473
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:33:57.929767 6345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:33:57.930020 6345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:33:57.930148 6345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:33:57.937197 6345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:33:57.937295 6345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:33:57.937438 6345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:33:57.946578 6345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:33:57.947726 6345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:33:57.948974 6345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:33:57.961334 6350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:57.962281 6351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:33:57.962481 6353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:33:57.963202 6345 server_base.cc:1034] running on GCE node
I20260619 04:33:57.963742 6345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:33:57.964660 6345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:33:57.965917 6345 hybrid_clock.cc:648] HybridClock initialized: now 1781843637965863 us; error 42 us; skew 500 ppm
I20260619 04:33:57.966241 6345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:33:57 dist-test-slave-n3tq krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:57.976239 6345 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:33:57.977645 6345 webserver.cc:460] Webserver started at http://127.0.97.3:39341/ using document root <none> and password file <none>
I20260619 04:33:57.978303 6345 fs_manager.cc:359] Metadata directory not provided
I20260619 04:33:57.978413 6345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:33:57.978679 6345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:33:57.981510 6345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "24bea2199508470d965c9afb33d3fe83"
format_stamp: "Formatted at 2026-06-19 04:33:57 on dist-test-slave-n3tq"
server_key: "65fd47fa2ac44f662082ae6799c5b475"
server_key_iv: "8c692164f235bf43a3b13ce3c9e44f00"
server_key_version: "encryptionkey@0"
I20260619 04:33:57.982240 6345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "24bea2199508470d965c9afb33d3fe83"
format_stamp: "Formatted at 2026-06-19 04:33:57 on dist-test-slave-n3tq"
server_key: "65fd47fa2ac44f662082ae6799c5b475"
server_key_iv: "8c692164f235bf43a3b13ce3c9e44f00"
server_key_version: "encryptionkey@0"
I20260619 04:33:57.986963 6345 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 04:33:57.990134 6360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:57.991408 6345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:33:57.991547 6345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "24bea2199508470d965c9afb33d3fe83"
format_stamp: "Formatted at 2026-06-19 04:33:57 on dist-test-slave-n3tq"
server_key: "65fd47fa2ac44f662082ae6799c5b475"
server_key_iv: "8c692164f235bf43a3b13ce3c9e44f00"
server_key_version: "encryptionkey@0"
I20260619 04:33:57.991720 6345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:33:58.008876 6345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:33:58.012568 6345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:33:58.012817 6345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:33:58.013680 6345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:33:58.014925 6345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:33:58.014994 6345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:58.015110 6345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:33:58.015161 6345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:33:58.043694 6345 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39205
I20260619 04:33:58.043746 6472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39205 every 8 connection(s)
I20260619 04:33:58.044904 6345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:33:58.053525 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 6345
I20260619 04:33:58.053691 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:33:58.054085 388 external_mini_cluster.cc:1428] Setting key 4fd76dd000ee654c0aa8844db3ef9e5f
Jun 19 04:33:58 dist-test-slave-n3tq krb5kdc[5972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:33:58.063158 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:58.047192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44329 (local address 127.0.97.62:43473)
0619 04:33:58.047351 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:58.047356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:58.048477 (+ 1121us) server_negotiation.cc:407] Connection header received
0619 04:33:58.050143 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:58.050148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:58.050223 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:58.050308 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:58.051748 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:58.052438 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:58.053982 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:58.054510 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:58.054631 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:58.058069 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:58.058073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:58.058077 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:58.058106 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:58.060532 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:58.061128 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:58.061134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:58.061138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:58.061260 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:58.061622 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:58.061627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:58.061630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:58.061891 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:58.062010 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:58.062633 (+ 623us) server_negotiation.cc:299] Negotiation successful
0619 04:33:58.062839 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 04:33:58.064025 6475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:58.047662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43473 (local address 127.0.97.3:44329)
0619 04:33:58.048313 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 04:33:58.048359 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 04:33:58.049900 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:33:58.050528 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:33:58.050536 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:33:58.051011 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:33:58.051596 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:58.051613 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:58.052598 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:58.052602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:58.053861 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:33:58.053873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:33:58.054724 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:33:58.054732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:33:58.055010 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:58.055629 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:33:58.055649 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:33:58.057880 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:33:58.060636 (+ 2756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:58.060645 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:58.060661 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:58.061026 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:58.061352 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:33:58.061357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:33:58.061361 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:33:58.061526 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:33:58.062001 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:33:58.062009 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:33:58.062314 (+ 305us) client_negotiation.cc:769] Sending connection context
0619 04:33:58.062540 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 04:33:58.062999 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":260,"threads_started":1}
I20260619 04:33:58.065251 6473 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43473
I20260619 04:33:58.065491 6473 heartbeater.cc:464] Registering TS with master...
I20260619 04:33:58.066123 6473 heartbeater.cc:511] Master 127.0.97.62:43473 requested a full tablet report, sending...
I20260619 04:33:58.067525 6017 ts_manager.cc:194] Registered new tserver with Master: 24bea2199508470d965c9afb33d3fe83 (127.0.97.3:39205)
I20260619 04:33:58.068486 6017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44329
I20260619 04:33:58.069072 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/test-admin.keytab.
Jun 19 04:33:58 dist-test-slave-n3tq krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:58.092726 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 04:33:58.097432 6480 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:33:58.095197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34336 (local address 127.0.97.1:42419)
0619 04:33:58.095797 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:58.095805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:58.095829 (+ 24us) server_negotiation.cc:407] Connection header received
0619 04:33:58.095887 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:58.095893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:58.096133 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:58.096291 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:58.097124 (+ 833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34336: BlockingRecv error: recv got EOF from 127.0.0.1:34336 (error 108)
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":215,"threads_started":1}
W20260619 04:33:58.097620 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/krb5cc' of type FILE
Jun 19 04:33:58 dist-test-slave-n3tq krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:58.099025 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 04:33:58 dist-test-slave-n3tq krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:58.100387 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 04:33:58 dist-test-slave-n3tq krb5kdc[5972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 04:33:58.114379 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:58.101904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34346 (local address 127.0.97.1:42419)
0619 04:33:58.102078 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:58.102084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:58.102226 (+ 142us) server_negotiation.cc:407] Connection header received
0619 04:33:58.102373 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:58.102376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:58.102433 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:58.102505 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:58.103423 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:58.104364 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:58.105599 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:58.106120 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:58.106211 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:58.109431 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:58.109441 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:58.109455 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:58.109485 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:58.111843 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:58.112443 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:58.112489 (+ 46us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:58.112493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:58.112618 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:58.112961 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:58.112966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:58.112970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:58.113305 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:58.113589 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:58.113886 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 04:33:58.114119 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 04:33:58.119517 6480 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:33:58.117980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34350 (local address 127.0.97.1:42419)
0619 04:33:58.118141 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:58.118146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:58.118407 (+ 261us) server_negotiation.cc:407] Connection header received
0619 04:33:58.118538 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:58.118543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:58.118609 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:58.118683 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:58.119338 (+ 655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34350: BlockingRecv error: recv got EOF from 127.0.0.1:34350 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 04:33:58.119846 388 init.cc:244] Error while opening credential cache '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/krb5cc)
Jun 19 04:33:58 dist-test-slave-n3tq krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:58.121198 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 04:33:58 dist-test-slave-n3tq krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:58.122289 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 04:33:58 dist-test-slave-n3tq krb5kdc[5972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 04:33:58.135259 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:58.123351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34352 (local address 127.0.97.1:42419)
0619 04:33:58.123484 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:58.123489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:58.123745 (+ 256us) server_negotiation.cc:407] Connection header received
0619 04:33:58.123866 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:58.123871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:58.123918 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:58.123988 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:58.124868 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:58.125640 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:58.126772 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:58.127208 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:58.127309 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:58.130826 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:58.130830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:58.130833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:58.130876 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:58.133443 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:58.134005 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:58.134009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:58.134011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:58.134115 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:58.134376 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:58.134378 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:58.134380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:58.134596 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:58.134769 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:58.134930 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 04:33:58.135063 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 04:33:58.138906 6480 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:33:58.137566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34356 (local address 127.0.97.1:42419)
0619 04:33:58.137724 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:58.137729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:58.137906 (+ 177us) server_negotiation.cc:407] Connection header received
0619 04:33:58.138002 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:58.138005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:58.138062 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:58.138129 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:58.138713 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34356: BlockingRecv error: recv got EOF from 127.0.0.1:34356 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 04:33:58.138999 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781843588881932-388-0/krb5kdc/krb5cc' of type FILE
Jun 19 04:33:58 dist-test-slave-n3tq krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:58.140357 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 04:33:58 dist-test-slave-n3tq krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:33:58.141532 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 04:33:58 dist-test-slave-n3tq krb5kdc[5972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 04:33:58.153855 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:58.142600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34358 (local address 127.0.97.1:42419)
0619 04:33:58.142761 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:58.142766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:58.142957 (+ 191us) server_negotiation.cc:407] Connection header received
0619 04:33:58.143054 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:58.143057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:58.143104 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:58.143173 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:58.143992 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:58.144744 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:58.145843 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:58.146206 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:58.146298 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:58.149399 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:58.149404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:58.149407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:58.149437 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:58.151658 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:58.152269 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:58.152273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:58.152275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:58.152446 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:58.152764 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:58.152767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:58.152769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:58.153021 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:58.153229 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:58.153411 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 04:33:58.153554 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 04:33:58.155141 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 6078
I20260619 04:33:58.156271 6206 heartbeater.cc:503] Master 127.0.97.62:43473 was elected leader, sending a full tablet report...
I20260619 04:33:58.165671 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:58.155754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34374 (local address 127.0.97.1:42419)
0619 04:33:58.155912 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:58.155918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:58.156225 (+ 307us) server_negotiation.cc:407] Connection header received
0619 04:33:58.156400 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:58.156405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:58.156498 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:58.156581 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:58.157543 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:58.158245 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:58.159324 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:58.159719 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:58.159821 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:58.161608 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:58.161612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:58.161615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:58.161640 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:58.163649 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:58.164204 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:58.164207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:58.164209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:58.164315 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:58.164614 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:58.164617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:58.164619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:58.164835 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:58.165007 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:58.165244 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 04:33:58.165375 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 04:33:58.166185 6201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:58.245767 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 6078
I20260619 04:33:58.258435 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 6212
Jun 19 04:33:58 dist-test-slave-n3tq krb5kdc[5972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 04:33:58.272403 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:58.259207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33010 (local address 127.0.97.2:44289)
0619 04:33:58.259773 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:58.259783 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:58.259818 (+ 35us) server_negotiation.cc:407] Connection header received
0619 04:33:58.259884 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:58.259891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:58.260100 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:58.260232 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:58.261253 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:58.262268 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:58.263342 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:58.263858 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:58.263951 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:58.267399 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:58.267409 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:58.267423 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:58.267458 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:58.269929 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:58.270539 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:58.270546 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:58.270548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:58.270662 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:58.271005 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:58.271009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:58.271011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:58.271316 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:58.271598 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:58.271879 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 04:33:58.272111 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":297,"threads_started":1}
I20260619 04:33:58.273571 6335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:58.353312 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 6212
I20260619 04:33:58.365430 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 6345
Jun 19 04:33:58 dist-test-slave-n3tq krb5kdc[5972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 04:33:58.380295 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:33:58.366328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36786 (local address 127.0.97.3:39205)
0619 04:33:58.366959 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0619 04:33:58.366966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:33:58.366993 (+ 27us) server_negotiation.cc:407] Connection header received
0619 04:33:58.367054 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:33:58.367060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:33:58.367290 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:33:58.367440 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:33:58.368448 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:58.369465 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:58.370662 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:33:58.371210 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:33:58.371325 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:33:58.375101 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:33:58.375112 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:33:58.375126 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:33:58.375158 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:33:58.377751 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:58.378384 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:58.378391 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:58.378393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:58.378506 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:33:58.378863 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:33:58.378867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:33:58.378869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:33:58.379191 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:33:58.379479 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0619 04:33:58.379755 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 04:33:58.379988 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":334,"threads_started":1}
I20260619 04:33:58.381517 6468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:58.462365 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 6345
I20260619 04:33:58.474552 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 5985
I20260619 04:33:58.475682 6047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:33:58.575852 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 5985
2026-06-19T04:33:58Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3553 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:33:58 dist-test-slave-n3tq krb5kdc[6500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:33:58 dist-test-slave-n3tq krb5kdc[6500](info): set up 2 sockets
Jun 19 04:33:58 dist-test-slave-n3tq krb5kdc[6500](info): commencing operation
krb5kdc: starting...
W20260619 04:33:59.568936 388 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:33:59 dist-test-slave-n3tq krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:33:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:33:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260619 04:33:59.749830 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42529
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:34:00.054133 6513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:00.054415 6513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:00.054529 6513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:00.061923 6513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:34:00.062042 6513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:00.062115 6513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:00.062187 6513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:34:00.062243 6513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:34:00.071910 6513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:00.073235 6513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:00.074488 6513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:00.085003 6518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:00.085035 6519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:00.085062 6521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:00.086494 6513 server_base.cc:1034] running on GCE node
I20260619 04:34:00.087249 6513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:00.088677 6513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:00.089874 6513 hybrid_clock.cc:648] HybridClock initialized: now 1781843640089820 us; error 48 us; skew 500 ppm
I20260619 04:34:00.090197 6513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:00 dist-test-slave-n3tq krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843640, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:00.093063 6513 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260619 04:34:00.094506 6513 webserver.cc:460] Webserver started at http://127.0.97.62:42201/ using document root <none> and password file <none>
I20260619 04:34:00.095135 6513 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:00.095227 6513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:00.095474 6513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:00.098331 6513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "71decb2899a44f1fb9b71a06ff359c0f"
format_stamp: "Formatted at 2026-06-19 04:34:00 on dist-test-slave-n3tq"
server_key: "59027d585f631f2e8be284d33bafe869"
server_key_iv: "7e78671a7ad9b6bfd701b59de83a6502"
server_key_version: "encryptionkey@0"
I20260619 04:34:00.099033 6513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "71decb2899a44f1fb9b71a06ff359c0f"
format_stamp: "Formatted at 2026-06-19 04:34:00 on dist-test-slave-n3tq"
server_key: "59027d585f631f2e8be284d33bafe869"
server_key_iv: "7e78671a7ad9b6bfd701b59de83a6502"
server_key_version: "encryptionkey@0"
I20260619 04:34:00.104115 6513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 04:34:00.107595 6528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:00.109016 6513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:34:00.109169 6513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "71decb2899a44f1fb9b71a06ff359c0f"
format_stamp: "Formatted at 2026-06-19 04:34:00 on dist-test-slave-n3tq"
server_key: "59027d585f631f2e8be284d33bafe869"
server_key_iv: "7e78671a7ad9b6bfd701b59de83a6502"
server_key_version: "encryptionkey@0"
I20260619 04:34:00.109349 6513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:00.137611 6513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:00.141281 6513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:00.141527 6513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:00.157380 6513 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42529
I20260619 04:34:00.157402 6579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42529 every 8 connection(s)
I20260619 04:34:00.158592 6513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:34:00.162000 6580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:00.166955 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 6513
I20260619 04:34:00.167164 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:34:00.167884 388 external_mini_cluster.cc:1428] Setting key 7328577275493504a1c8aef91185c243
I20260619 04:34:00.172590 6580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f: Bootstrap starting.
I20260619 04:34:00.175915 6580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:00.177098 6580 log.cc:824] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f: Log is configured to *not* fsync() on all Append() calls
I20260619 04:34:00.180161 6580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f: No bootstrap required, opened a new log
Jun 19 04:34:00 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 04:34:00.187592 6580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } }
I20260619 04:34:00.187923 6580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:00.188024 6580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71decb2899a44f1fb9b71a06ff359c0f, State: Initialized, Role: FOLLOWER
I20260619 04:34:00.188638 6580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } }
I20260619 04:34:00.188853 6580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:34:00.188979 6580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:34:00.189138 6580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:00.189513 6583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:00.169840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57960 (local address 127.0.97.62:42529)
0619 04:34:00.170615 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:00.170626 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:00.170669 (+ 43us) server_negotiation.cc:407] Connection header received
0619 04:34:00.172215 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:00.172239 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:00.172647 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:00.172966 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:00.174566 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:00.175754 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:00.177448 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:00.178430 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:00.178548 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:00.182736 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:00.182744 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:00.182758 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:00.182790 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:00.185637 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:00.186239 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:00.186246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:00.186252 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:00.186424 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:00.186756 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:00.186759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:00.186761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:00.187258 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:00.187545 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:00.187990 (+ 445us) server_negotiation.cc:299] Negotiation successful
0619 04:34:00.188420 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":359,"threads_started":1}
I20260619 04:34:00.191313 6580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } }
I20260619 04:34:00.191754 6580 leader_election.cc:302] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71decb2899a44f1fb9b71a06ff359c0f; no voters:
I20260619 04:34:00.192354 6580 leader_election.cc:288] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:34:00.192418 6585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:34:00.193039 6585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [term 1 LEADER]: Becoming Leader. State: Replica: 71decb2899a44f1fb9b71a06ff359c0f, State: Running, Role: LEADER
I20260619 04:34:00.193488 6585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } }
I20260619 04:34:00.194212 6580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:34:00.195997 6586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71decb2899a44f1fb9b71a06ff359c0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } }
I20260619 04:34:00.196272 6586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [sys.catalog]: This master's current role is: LEADER
I20260619 04:34:00.196694 6587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71decb2899a44f1fb9b71a06ff359c0f. Latest consensus state: current_term: 1 leader_uuid: "71decb2899a44f1fb9b71a06ff359c0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } }
I20260619 04:34:00.197059 6587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [sys.catalog]: This master's current role is: LEADER
I20260619 04:34:00.200958 6590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:34:00.206079 6590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:34:00.213992 6590 catalog_manager.cc:1347] Generated new cluster ID: 7704a7aa6f7e4cddaf287d60a6d4102c
I20260619 04:34:00.214102 6590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:34:00.232344 6590 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:34:00.233610 6590 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:34:00.247388 6590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f: Generated new TSK 0
I20260619 04:34:00.248178 6590 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260619 04:34:00.299584 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42529
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:34:00.609506 6606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:00.609871 6606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:00.610023 6606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:00.618278 6606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:00.618382 6606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:00.618533 6606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:34:00.628059 6606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:00.629307 6606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:00.630597 6606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:00.642493 6614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:00.642547 6612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:00.642565 6611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:00.643249 6606 server_base.cc:1034] running on GCE node
I20260619 04:34:00.643819 6606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:00.644764 6606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:00.645995 6606 hybrid_clock.cc:648] HybridClock initialized: now 1781843640645950 us; error 30 us; skew 500 ppm
I20260619 04:34:00.646324 6606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:00 dist-test-slave-n3tq krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843640, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:00.649255 6606 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 04:34:00.650540 6606 webserver.cc:460] Webserver started at http://127.0.97.1:34785/ using document root <none> and password file <none>
I20260619 04:34:00.651175 6606 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:00.651279 6606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:00.651566 6606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:00.654369 6606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "6af077b208f048a2bc017df0886b3e70"
format_stamp: "Formatted at 2026-06-19 04:34:00 on dist-test-slave-n3tq"
server_key: "71ee397c4b6c8c23b0f6cd6a66bea4a4"
server_key_iv: "d5b075179135f7b28c6b6374248e6b90"
server_key_version: "encryptionkey@0"
I20260619 04:34:00.655085 6606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6af077b208f048a2bc017df0886b3e70"
format_stamp: "Formatted at 2026-06-19 04:34:00 on dist-test-slave-n3tq"
server_key: "71ee397c4b6c8c23b0f6cd6a66bea4a4"
server_key_iv: "d5b075179135f7b28c6b6374248e6b90"
server_key_version: "encryptionkey@0"
I20260619 04:34:00.660281 6606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 04:34:00.663456 6621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:00.664744 6606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:34:00.664880 6606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "6af077b208f048a2bc017df0886b3e70"
format_stamp: "Formatted at 2026-06-19 04:34:00 on dist-test-slave-n3tq"
server_key: "71ee397c4b6c8c23b0f6cd6a66bea4a4"
server_key_iv: "d5b075179135f7b28c6b6374248e6b90"
server_key_version: "encryptionkey@0"
I20260619 04:34:00.665055 6606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:00.695514 6606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:00.699782 6606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:00.700047 6606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:00.701035 6606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:34:00.702296 6606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:00.702369 6606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:00.702498 6606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:00.702557 6606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:00.730340 6606 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38759
I20260619 04:34:00.730382 6733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38759 every 8 connection(s)
I20260619 04:34:00.731554 6606 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:34:00.737512 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 6606
I20260619 04:34:00.737743 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:34:00.738487 388 external_mini_cluster.cc:1428] Setting key 5bc413566146a6099adce7404c948e8e
Jun 19 04:34:00 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843640, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 04:34:00.752169 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:00.733921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44221 (local address 127.0.97.62:42529)
0619 04:34:00.734316 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:00.734322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:00.735264 (+ 942us) server_negotiation.cc:407] Connection header received
0619 04:34:00.737031 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:00.737036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:00.737100 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:00.737192 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:00.739157 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:00.740117 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:00.742263 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:00.742647 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:00.742743 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:00.746473 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:00.746479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:00.746482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:00.746514 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:00.749236 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:00.749967 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:00.749971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:00.749973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:00.750086 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:00.750492 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:00.750496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:00.750499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:00.750770 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:00.750894 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:00.751591 (+ 697us) server_negotiation.cc:299] Negotiation successful
0619 04:34:00.751813 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":180,"threads_started":1}
I20260619 04:34:00.753206 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:00.734472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.97.1:44221)
0619 04:34:00.735108 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:00.735149 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:00.736787 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:00.737486 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:00.737499 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:00.738192 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:00.738970 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:00.738993 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:00.740264 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:00.740268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:00.742132 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:00.742146 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:00.742851 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:00.742857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:00.743119 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:00.743765 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:00.743790 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:00.746222 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:00.749392 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:00.749402 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:00.749422 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:00.749841 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:00.750188 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:00.750193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:00.750197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:00.750380 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:00.750887 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:00.750896 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:00.751244 (+ 348us) client_negotiation.cc:769] Sending connection context
0619 04:34:00.751484 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 04:34:00.752021 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":247,"threads_started":1}
I20260619 04:34:00.754899 6734 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42529
I20260619 04:34:00.755185 6734 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:00.755890 6734 heartbeater.cc:511] Master 127.0.97.62:42529 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 04:34:00.758226 6545 ts_manager.cc:194] Registered new tserver with Master: 6af077b208f048a2bc017df0886b3e70 (127.0.97.1:38759)
I20260619 04:34:00.759776 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260619 04:34:00.782847 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42529
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:34:01.089066 6740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:01.089327 6740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:01.089458 6740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:01.096629 6740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:01.096735 6740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:01.096884 6740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:34:01.106508 6740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:01.107724 6740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:01.109066 6740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:01.121469 6745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:01.121533 6746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:01.121501 6748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:01.122319 6740 server_base.cc:1034] running on GCE node
I20260619 04:34:01.122905 6740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:01.123832 6740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:01.125020 6740 hybrid_clock.cc:648] HybridClock initialized: now 1781843641124970 us; error 41 us; skew 500 ppm
I20260619 04:34:01.125316 6740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:01 dist-test-slave-n3tq krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843641, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:01.128170 6740 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 04:34:01.129560 6740 webserver.cc:460] Webserver started at http://127.0.97.2:33225/ using document root <none> and password file <none>
I20260619 04:34:01.130208 6740 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:01.130317 6740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:01.130578 6740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:01.133366 6740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "6f5a5fd7474244c6bfb857386b7e9e0e"
format_stamp: "Formatted at 2026-06-19 04:34:01 on dist-test-slave-n3tq"
server_key: "6f1f84a02f63aced7837a961bbd8e9f4"
server_key_iv: "52eb3ed070e18c6705c95b136f51a94a"
server_key_version: "encryptionkey@0"
I20260619 04:34:01.134097 6740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6f5a5fd7474244c6bfb857386b7e9e0e"
format_stamp: "Formatted at 2026-06-19 04:34:01 on dist-test-slave-n3tq"
server_key: "6f1f84a02f63aced7837a961bbd8e9f4"
server_key_iv: "52eb3ed070e18c6705c95b136f51a94a"
server_key_version: "encryptionkey@0"
I20260619 04:34:01.139057 6740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 04:34:01.142416 6755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:01.143682 6740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:34:01.143831 6740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "6f5a5fd7474244c6bfb857386b7e9e0e"
format_stamp: "Formatted at 2026-06-19 04:34:01 on dist-test-slave-n3tq"
server_key: "6f1f84a02f63aced7837a961bbd8e9f4"
server_key_iv: "52eb3ed070e18c6705c95b136f51a94a"
server_key_version: "encryptionkey@0"
I20260619 04:34:01.144002 6740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:01.160236 6740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:01.164361 6740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:01.164633 6740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:01.165539 6740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:34:01.166775 6740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:01.166846 6740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:01.166958 6740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:01.167007 6740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:01.195407 6740 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37011
I20260619 04:34:01.195463 6867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37011 every 8 connection(s)
I20260619 04:34:01.196704 6740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:34:01.199826 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 6740
I20260619 04:34:01.200003 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:34:01.200445 388 external_mini_cluster.cc:1428] Setting key 4535ae8a054986c7521d834b91f2c3de
Jun 19 04:34:01 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843641, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 04:34:01.216429 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:01.199086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40407 (local address 127.0.97.62:42529)
0619 04:34:01.199248 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:01.199252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:01.200515 (+ 1263us) server_negotiation.cc:407] Connection header received
0619 04:34:01.202406 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:01.202412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:01.202502 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:01.202608 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:01.203970 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:01.204916 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:01.206603 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:01.207038 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:01.207150 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:01.210738 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:01.210743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:01.210747 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:01.210779 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:01.213328 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:01.214209 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:01.214213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:01.214215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:01.214337 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:01.214761 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:01.214764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:01.214766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:01.215014 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:01.215139 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:01.215860 (+ 721us) server_negotiation.cc:299] Negotiation successful
0619 04:34:01.216082 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 04:34:01.217438 6870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:01.199567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.97.2:40407)
0619 04:34:01.200320 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:01.200365 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:01.202126 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:01.202779 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:01.202787 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:01.203266 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:01.203797 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:01.203813 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:01.205085 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:01.205089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:01.206437 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:01.206451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:01.207248 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:01.207255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:01.207482 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:01.208160 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:01.208183 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:01.210519 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:01.213475 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:01.213484 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:01.213501 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:01.214108 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:01.214435 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:01.214439 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:01.214442 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:01.214672 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:01.215137 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:01.215146 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:01.215487 (+ 341us) client_negotiation.cc:769] Sending connection context
0619 04:34:01.215758 (+ 271us) client_negotiation.cc:240] Negotiation successful
0619 04:34:01.216314 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":309,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 04:34:01.218982 6868 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42529
I20260619 04:34:01.219262 6868 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:01.220000 6868 heartbeater.cc:511] Master 127.0.97.62:42529 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
I20260619 04:34:01.221488 6545 ts_manager.cc:194] Registered new tserver with Master: 6f5a5fd7474244c6bfb857386b7e9e0e (127.0.97.2:37011)
I20260619 04:34:01.222259 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260619 04:34:01.243533 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42529
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:34:01.553434 6873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:01.553723 6873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:01.553838 6873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:01.561058 6873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:01.561170 6873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:01.561313 6873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:34:01.571108 6873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:01.572368 6873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:01.573655 6873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:01.585875 6878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:01.585943 6879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:01.585947 6881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:01.586724 6873 server_base.cc:1034] running on GCE node
I20260619 04:34:01.587275 6873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:01.588176 6873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:01.589372 6873 hybrid_clock.cc:648] HybridClock initialized: now 1781843641589288 us; error 72 us; skew 500 ppm
I20260619 04:34:01.589676 6873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:01 dist-test-slave-n3tq krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843641, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:01.592581 6873 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 04:34:01.593891 6873 webserver.cc:460] Webserver started at http://127.0.97.3:46745/ using document root <none> and password file <none>
I20260619 04:34:01.594624 6873 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:01.594733 6873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:01.595019 6873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:01.597889 6873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "7ec7bb13dce74266a393d0693e2b6ff5"
format_stamp: "Formatted at 2026-06-19 04:34:01 on dist-test-slave-n3tq"
server_key: "93d446785745a6a1c5ee801eb7ed8752"
server_key_iv: "8c782012d203074d9b09bad71367dfc6"
server_key_version: "encryptionkey@0"
I20260619 04:34:01.598614 6873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7ec7bb13dce74266a393d0693e2b6ff5"
format_stamp: "Formatted at 2026-06-19 04:34:01 on dist-test-slave-n3tq"
server_key: "93d446785745a6a1c5ee801eb7ed8752"
server_key_iv: "8c782012d203074d9b09bad71367dfc6"
server_key_version: "encryptionkey@0"
I20260619 04:34:01.603470 6873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 04:34:01.606781 6888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:01.608104 6873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:34:01.608245 6873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "7ec7bb13dce74266a393d0693e2b6ff5"
format_stamp: "Formatted at 2026-06-19 04:34:01 on dist-test-slave-n3tq"
server_key: "93d446785745a6a1c5ee801eb7ed8752"
server_key_iv: "8c782012d203074d9b09bad71367dfc6"
server_key_version: "encryptionkey@0"
I20260619 04:34:01.608414 6873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:01.638162 6873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:01.642303 6873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:01.642650 6873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:01.643601 6873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:34:01.644917 6873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:01.644989 6873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:01.645098 6873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:01.645152 6873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:01.673434 6873 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42411
I20260619 04:34:01.673481 7000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42411 every 8 connection(s)
I20260619 04:34:01.674633 6873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:34:01.681591 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 6873
I20260619 04:34:01.681767 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:34:01.682158 388 external_mini_cluster.cc:1428] Setting key b9fe6c527d6f8c8befc4aa349dc7ad78
Jun 19 04:34:01 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843641, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 04:34:01.692860 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:01.677041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53039 (local address 127.0.97.62:42529)
0619 04:34:01.677208 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:01.677212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:01.678302 (+ 1090us) server_negotiation.cc:407] Connection header received
0619 04:34:01.680050 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:01.680056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:01.680135 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:01.680223 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:01.681710 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:01.682504 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:01.684114 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:01.684583 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:01.684696 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:01.687733 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:01.687738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:01.687741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:01.687771 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:01.690026 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:01.690684 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:01.690690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:01.690693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:01.690803 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:01.691188 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:01.691194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:01.691197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:01.691477 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:01.691602 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:01.692270 (+ 668us) server_negotiation.cc:299] Negotiation successful
0619 04:34:01.692525 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 04:34:01.693778 7003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:01.677500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.97.3:53039)
0619 04:34:01.678143 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:01.678183 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:01.679775 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:01.680492 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:01.680503 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:01.680996 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:01.681548 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:01.681564 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:01.682653 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:01.682657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:01.683984 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:01.683999 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:01.684777 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:01.684785 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:01.685029 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:01.685567 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:01.685588 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:01.687536 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:01.690163 (+ 2627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:01.690172 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:01.690190 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:01.690578 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:01.690900 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:01.690905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:01.690909 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:01.691089 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:01.691594 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:01.691601 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:01.691920 (+ 319us) client_negotiation.cc:769] Sending connection context
0619 04:34:01.692171 (+ 251us) client_negotiation.cc:240] Negotiation successful
0619 04:34:01.692714 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":248,"threads_started":1}
I20260619 04:34:01.695207 7001 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42529
I20260619 04:34:01.695550 7001 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:01.696429 7001 heartbeater.cc:511] Master 127.0.97.62:42529 requested a full tablet report, sending...
I20260619 04:34:01.697857 6545 ts_manager.cc:194] Registered new tserver with Master: 7ec7bb13dce74266a393d0693e2b6ff5 (127.0.97.3:42411)
I20260619 04:34:01.698689 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53039
I20260619 04:34:01.709650 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260619 04:34:01.755823 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39847
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42529,127.0.97.61:39847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:34:01.763375 6734 heartbeater.cc:503] Master 127.0.97.62:42529 was elected leader, sending a full tablet report...
W20260619 04:34:02.070489 7006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:02.070793 7006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:02.070933 7006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:02.078178 7006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:34:02.078307 7006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:02.078379 7006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:02.078451 7006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:34:02.078513 7006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:34:02.088474 7006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42529,127.0.97.61:39847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
W20260619 04:34:02.088836 7006 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:42529,127.0.97.61:39847'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 04:34:02.089855 7006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:02.091135 7006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:02.102487 7011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:02.102557 7014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:02.102584 7012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:02.103623 7006 server_base.cc:1034] running on GCE node
I20260619 04:34:02.104367 7006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:02.105818 7006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:02.107044 7006 hybrid_clock.cc:648] HybridClock initialized: now 1781843642106946 us; error 58 us; skew 500 ppm
I20260619 04:34:02.107331 7006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:02 dist-test-slave-n3tq krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843642, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:02.110311 7006 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260619 04:34:02.111714 7006 webserver.cc:460] Webserver started at http://127.0.97.61:40117/ using document root <none> and password file <none>
I20260619 04:34:02.112409 7006 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:02.112537 7006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:02.112895 7006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:02.115909 7006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/data/instance:
uuid: "7a74506f385f403d9104e5dcdb2c851a"
format_stamp: "Formatted at 2026-06-19 04:34:02 on dist-test-slave-n3tq"
server_key: "cc29f598997ab5b22ea9e838eafd8691"
server_key_iv: "139816cd86ef6bf0423f8b312a48b1a6"
server_key_version: "encryptionkey@0"
I20260619 04:34:02.116737 7006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/wal/instance:
uuid: "7a74506f385f403d9104e5dcdb2c851a"
format_stamp: "Formatted at 2026-06-19 04:34:02 on dist-test-slave-n3tq"
server_key: "cc29f598997ab5b22ea9e838eafd8691"
server_key_iv: "139816cd86ef6bf0423f8b312a48b1a6"
server_key_version: "encryptionkey@0"
I20260619 04:34:02.121927 7006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 04:34:02.125353 7021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:02.126786 7006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:34:02.126964 7006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/wal
uuid: "7a74506f385f403d9104e5dcdb2c851a"
format_stamp: "Formatted at 2026-06-19 04:34:02 on dist-test-slave-n3tq"
server_key: "cc29f598997ab5b22ea9e838eafd8691"
server_key_iv: "139816cd86ef6bf0423f8b312a48b1a6"
server_key_version: "encryptionkey@0"
I20260619 04:34:02.127166 7006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:02.143177 7006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:02.147315 7006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:02.147557 7006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:02.164081 7006 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:39847
I20260619 04:34:02.164105 7072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:39847 every 8 connection(s)
I20260619 04:34:02.165596 7006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/data/info.pb
I20260619 04:34:02.169162 7073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:02.174623 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 7006
I20260619 04:34:02.175279 7073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:34:02 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843642, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 19 04:34:02 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 04:34:02.193581 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.177919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57972 (local address 127.0.97.62:42529)
0619 04:34:02.178100 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:02.178106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:02.178814 (+ 708us) server_negotiation.cc:407] Connection header received
0619 04:34:02.179588 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:02.179593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:02.179665 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:02.179762 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:02.180936 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.181891 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.183859 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.184529 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.184656 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.187836 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:02.187844 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:02.187848 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:02.187890 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:02.190773 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.191466 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.191471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.191475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.191586 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.191946 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.191951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.191955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.192245 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:02.192366 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:02.193075 (+ 709us) server_negotiation.cc:299] Negotiation successful
0619 04:34:02.193271 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 04:34:02.194535 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.177899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.0.1:57972)
0619 04:34:02.178663 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:02.178709 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:02.179322 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:02.179796 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:02.179801 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:02.180387 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:02.180777 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.180793 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.182017 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.182022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.183741 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.183750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.184703 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.184707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.184913 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.185604 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:02.185633 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:02.187649 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:02.190881 (+ 3232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.190892 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.190904 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.191366 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.191681 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.191686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.191693 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.191851 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.192360 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:02.192368 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:02.192705 (+ 337us) client_negotiation.cc:769] Sending connection context
0619 04:34:02.192961 (+ 256us) client_negotiation.cc:240] Negotiation successful
0619 04:34:02.193412 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":17,"thread_start_us":370,"threads_started":1}
I20260619 04:34:02.195261 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.176713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43348 (local address 127.0.97.61:39847)
0619 04:34:02.177417 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:02.177428 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:02.177464 (+ 36us) server_negotiation.cc:407] Connection header received
0619 04:34:02.179317 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:02.179338 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:02.179604 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:02.179807 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:02.181730 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.183379 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.184883 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.185604 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.185724 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.189339 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:02.189349 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:02.189364 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:02.189404 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:02.192176 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.192829 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.192835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.192837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.192964 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.193295 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.193298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.193300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.193593 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:02.193859 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:02.194601 (+ 742us) server_negotiation.cc:299] Negotiation successful
0619 04:34:02.194881 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":516,"thread_start_us":313,"threads_started":1}
I20260619 04:34:02.196509 7073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:34:02 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843642, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 04:34:02.208220 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.197299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39847 (local address 127.0.0.1:43354)
0619 04:34:02.197553 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:02.197570 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:02.197660 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:02.198050 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:02.198054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:02.198336 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:02.198486 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.198494 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.199777 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.199780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.200708 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.200719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.201296 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.201300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.201446 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.201871 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:02.201885 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:02.203567 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:02.206174 (+ 2607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.206178 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.206180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.206488 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.206775 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.206778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.206780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.206886 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.207464 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:02.207467 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:02.207578 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 04:34:02.207672 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 04:34:02.207862 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260619 04:34:02.208298 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.197213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43354 (local address 127.0.97.61:39847)
0619 04:34:02.197382 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:02.197387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:02.197737 (+ 350us) server_negotiation.cc:407] Connection header received
0619 04:34:02.197770 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:02.197774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:02.197830 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:02.197900 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:02.198607 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.199658 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.200813 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.201182 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.201269 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.203734 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:02.203739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:02.203742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:02.203769 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:02.206068 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.206584 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.206588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.206590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.206696 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.206995 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.206999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.207001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.207364 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:02.207473 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:02.207754 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 04:34:02.208023 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":54}
I20260619 04:34:02.212649 7073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a: Bootstrap starting.
I20260619 04:34:02.215626 7073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:02.216601 7073 log.cc:824] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a: Log is configured to *not* fsync() on all Append() calls
I20260619 04:34:02.218997 7073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a: No bootstrap required, opened a new log
I20260619 04:34:02.224664 6868 heartbeater.cc:503] Master 127.0.97.62:42529 was elected leader, sending a full tablet report...
I20260619 04:34:02.225524 7073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } }
I20260619 04:34:02.225807 7073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:02.225916 7073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a74506f385f403d9104e5dcdb2c851a, State: Initialized, Role: FOLLOWER
I20260619 04:34:02.226480 7073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } }
I20260619 04:34:02.227209 7079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } } }
I20260619 04:34:02.227483 7079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:34:02.228108 7073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [sys.catalog]: configured and running, proceeding with master startup.
W20260619 04:34:02.236341 7090 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 04:34:02.236517 7090 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 04:34:02.238101 7006 master_runner.cc:428] Detected that this master 7a74506f385f403d9104e5dcdb2c851a is joining an existing cluster
I20260619 04:34:02.238183 7006 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:39847
I20260619 04:34:02.256102 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.242104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57982 (local address 127.0.97.62:42529)
0619 04:34:02.242264 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:02.242269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:02.242473 (+ 204us) server_negotiation.cc:407] Connection header received
0619 04:34:02.242723 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:02.242729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:02.242798 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:02.242876 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:02.243847 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.244671 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.246587 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.247063 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.247221 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.249767 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:02.249773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:02.249778 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:02.249816 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:02.253034 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.253746 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.253751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.253755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.253925 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.254399 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.254404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.254408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.254753 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:02.254892 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:02.255537 (+ 645us) server_negotiation.cc:299] Negotiation successful
0619 04:34:02.255745 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:34:02.257278 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.241722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.0.1:57982)
0619 04:34:02.242376 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:02.242395 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:02.242593 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:02.242927 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:02.242931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:02.243411 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:02.243715 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.243724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.244838 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.244843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.246414 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.246433 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.247255 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.247261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.247569 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.248407 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:02.248430 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:02.249577 (+ 1147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:02.253175 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.253180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.253183 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.253614 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.254073 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.254077 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.254081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.254247 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.254953 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:02.254958 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:02.255161 (+ 203us) client_negotiation.cc:769] Sending connection context
0619 04:34:02.256537 (+ 1376us) client_negotiation.cc:240] Negotiation successful
0619 04:34:02.256823 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":538,"threads_started":1}
I20260619 04:34:02.258774 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.241722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43370 (local address 127.0.97.61:39847)
0619 04:34:02.241944 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:02.241951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:02.245066 (+ 3115us) server_negotiation.cc:407] Connection header received
0619 04:34:02.245218 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:02.245232 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:02.245306 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:02.245407 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:02.247056 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.247979 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.249854 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.250538 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.250618 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.252733 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:02.252739 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:02.252744 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:02.252775 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:02.256050 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.256819 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.256824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.256827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.256975 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.257370 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.257375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.257379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.257718 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:02.257833 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:02.258252 (+ 419us) server_negotiation.cc:299] Negotiation successful
0619 04:34:02.258444 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 04:34:02.259742 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.241941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39847 (local address 127.0.0.1:43370)
0619 04:34:02.244961 (+ 3020us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:02.244978 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:02.245103 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:02.245445 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:02.245450 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:02.246100 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:02.246948 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.246958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.248167 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.248171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.249733 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.249744 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.250875 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.250885 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.251049 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.251581 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:02.251602 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:02.252566 (+ 964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:02.256201 (+ 3635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.256206 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.256210 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.256702 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.257099 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.257103 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.257106 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.257254 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.257903 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:02.257907 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:02.258029 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 04:34:02.259077 (+ 1048us) client_negotiation.cc:240] Negotiation successful
0619 04:34:02.259346 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2867,"thread_start_us":2674,"threads_started":1}
Jun 19 04:34:02 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843640, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 04:34:02.280057 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.263557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.0.97.61:39847)
0619 04:34:02.263718 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:02.263724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:02.264260 (+ 536us) server_negotiation.cc:407] Connection header received
0619 04:34:02.264556 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:02.264570 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:02.264687 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:02.264792 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:02.265737 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.266827 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.268091 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.268848 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.268972 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.273054 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:02.273060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:02.273065 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:02.273105 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:02.276581 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.277583 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.277589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.277593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.277757 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.278255 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.278260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.278265 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.278646 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:02.278795 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:02.279426 (+ 631us) server_negotiation.cc:299] Negotiation successful
0619 04:34:02.279654 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 04:34:02.280107 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.263489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39847 (local address 127.0.0.1:43382)
0619 04:34:02.264122 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:02.264157 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:02.264349 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:02.264822 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:02.264827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:02.265418 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:02.265608 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.265617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.266938 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.266942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.267964 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.267975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.269013 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.269019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.269251 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.270038 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:02.270058 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:02.272841 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:02.276756 (+ 3915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.276772 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.276782 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.277398 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.277933 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.277938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.277942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.278107 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.278825 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:02.278834 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:02.279050 (+ 216us) client_negotiation.cc:769] Sending connection context
0619 04:34:02.279302 (+ 252us) client_negotiation.cc:240] Negotiation successful
0619 04:34:02.279674 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":265,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260619 04:34:02.282454 6545 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:39847: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 39847
}
attrs {
promote: true
}
}
dest_uuid: "71decb2899a44f1fb9b71a06ff359c0f"
cas_config_opid_index: -1
I20260619 04:34:02.283454 6545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
I20260619 04:34:02.291379 7048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:02.291043 7102 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7a74506f385f403d9104e5dcdb2c851a (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: true } } }
I20260619 04:34:02.294868 7048 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 1 FOLLOWER]: Refusing update from remote peer 71decb2899a44f1fb9b71a06ff359c0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 04:34:02.295415 7102 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:39847
I20260619 04:34:02.295998 7102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:34:02.296203 7006 master.cc:537] Master@127.0.97.61:39847 shutting down...
I20260619 04:34:02.298028 7101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "71decb2899a44f1fb9b71a06ff359c0f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: true } } }
I20260619 04:34:02.298451 7101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [sys.catalog]: This master's current role is: LEADER
W20260619 04:34:02.299309 6532 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 04:34:02.302780 6532 consensus_peers.cc:487] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f -> Peer 7a74506f385f403d9104e5dcdb2c851a (127.0.97.61:39847): Couldn't send request to peer 7a74506f385f403d9104e5dcdb2c851a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 04:34:02.307863 7006 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:34:02.308264 7006 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:34:02.308477 7006 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a: stopping tablet replica
I20260619 04:34:02.314348 7006 master.cc:552] Master@127.0.97.61:39847 shutdown complete.
I20260619 04:34:02.314414 7006 master_runner.cc:305] Clearing existing system tablet
I20260619 04:34:02.314749 7006 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:02.314819 7006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:02.317066 7006 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 04:34:02.318419 7107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:02.318696 7006 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:02.318938 7006 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 04:34:02.319012 7006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/wal
uuid: "7a74506f385f403d9104e5dcdb2c851a"
format_stamp: "Formatted at 2026-06-19 04:34:02 on dist-test-slave-n3tq"
server_key: "cc29f598997ab5b22ea9e838eafd8691"
server_key_iv: "139816cd86ef6bf0423f8b312a48b1a6"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260619 04:34:02.322409 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39307
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42529,127.0.97.61:39847,127.0.97.60:39307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:34:02.322970 7006 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:02.329345 7006 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 04:34:02.329511 7006 log.cc:1196] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 04:34:02.329876 7006 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a: Deleting consensus metadata
I20260619 04:34:02.331655 7006 master_runner.cc:315] Copying system tablet from 127.0.97.62:42529
I20260619 04:34:02.333877 7006 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:42529
I20260619 04:34:02.345801 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.334620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57984 (local address 127.0.97.62:42529)
0619 04:34:02.334834 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:02.334841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:02.335288 (+ 447us) server_negotiation.cc:407] Connection header received
0619 04:34:02.335461 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:02.335467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:02.335549 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:02.335641 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:02.336538 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.337230 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.338465 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.339055 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.339178 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.340956 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:02.340963 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:02.340967 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:02.341003 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:02.343370 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.343996 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.344001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.344004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.344124 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.344471 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.344477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.344480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.344763 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:02.344898 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:02.345253 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 04:34:02.345449 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 04:34:02.345903 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.334567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.0.1:57984)
0619 04:34:02.335179 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:02.335194 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:02.335342 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:02.335681 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:02.335685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:02.336101 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:02.336303 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.336316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.337369 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.337373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.338334 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.338346 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.339197 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.339203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.339365 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.339950 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:02.339967 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:02.340680 (+ 713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:02.343505 (+ 2825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.343511 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.343516 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.343869 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.344221 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.344226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.344230 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.344355 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.344871 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:02.344875 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:02.345029 (+ 154us) client_negotiation.cc:769] Sending connection context
0619 04:34:02.345155 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 04:34:02.345449 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":378,"threads_started":1}
I20260619 04:34:02.346918 6565 tablet_copy_service.cc:140] P 71decb2899a44f1fb9b71a06ff359c0f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7a74506f385f403d9104e5dcdb2c851a ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:57984)
I20260619 04:34:02.347142 6565 tablet_copy_service.cc:161] P 71decb2899a44f1fb9b71a06ff359c0f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7a74506f385f403d9104e5dcdb2c851a at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:57984: session id = 7a74506f385f403d9104e5dcdb2c851a-00000000000000000000000000000000
I20260619 04:34:02.350274 6565 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 04:34:02.352125 7006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 04:34:02.356657 7006 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a: tablet copy: Starting download of 0 data blocks...
I20260619 04:34:02.356921 7006 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a: tablet copy: Starting download of 1 WAL segments...
I20260619 04:34:02.359215 7006 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 04:34:02.362367 6565 tablet_copy_service.cc:342] P 71decb2899a44f1fb9b71a06ff359c0f: Request end of tablet copy session 7a74506f385f403d9104e5dcdb2c851a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:57984
I20260619 04:34:02.362538 6565 tablet_copy_service.cc:433] P 71decb2899a44f1fb9b71a06ff359c0f: ending tablet copy session 7a74506f385f403d9104e5dcdb2c851a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7a74506f385f403d9104e5dcdb2c851a
I20260619 04:34:02.372530 7006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:02.374958 7115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:02.381503 7116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:02.383940 7006 server_base.cc:1034] running on GCE node
W20260619 04:34:02.388741 7118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:02.389206 7006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:02.389807 7006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:02.391011 7006 hybrid_clock.cc:648] HybridClock initialized: now 1781843642390956 us; error 33 us; skew 500 ppm
I20260619 04:34:02.391224 7006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:02 dist-test-slave-n3tq krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843642, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:02.393018 7006 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260619 04:34:02.394232 7006 webserver.cc:460] Webserver started at http://127.0.97.61:46323/ using document root <none> and password file <none>
I20260619 04:34:02.394541 7006 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:02.394652 7006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:02.397143 7006 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 04:34:02.398623 7125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:02.399021 7006 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 04:34:02.399145 7006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/wal
uuid: "7a74506f385f403d9104e5dcdb2c851a"
format_stamp: "Formatted at 2026-06-19 04:34:02 on dist-test-slave-n3tq"
server_key: "cc29f598997ab5b22ea9e838eafd8691"
server_key_iv: "139816cd86ef6bf0423f8b312a48b1a6"
server_key_version: "encryptionkey@0"
I20260619 04:34:02.399286 7006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:02.415413 7006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:02.418511 7006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:02.445649 7006 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:39847
I20260619 04:34:02.445668 7176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:39847 every 8 connection(s)
I20260619 04:34:02.446527 7006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-1/data/info.pb
I20260619 04:34:02.453778 7177 sys_catalog.cc:263] Verifying existing consensus state
I20260619 04:34:02.458478 7177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a: Bootstrap starting.
I20260619 04:34:02.478154 7177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 04:34:02.478701 7177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a: Bootstrap complete.
I20260619 04:34:02.479749 7177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: true } }
I20260619 04:34:02.480023 7177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a74506f385f403d9104e5dcdb2c851a, State: Initialized, Role: LEARNER
I20260619 04:34:02.480253 7177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: true } }
I20260619 04:34:02.482031 7180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: true } } }
I20260619 04:34:02.482304 7177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:34:02.482528 7180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [sys.catalog]: This master's current role is: LEARNER
I20260619 04:34:02.498315 7191 catalog_manager.cc:1259] Loaded cluster ID: 7704a7aa6f7e4cddaf287d60a6d4102c
I20260619 04:34:02.498584 7191 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a: loading cluster ID for follower catalog manager: success
I20260619 04:34:02.502221 7191 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a: acquiring CA information for follower catalog manager: success
I20260619 04:34:02.504410 7191 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 04:34:02.696611 7108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:02.696892 7108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:02.697017 7108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 04:34:02.701304 7001 heartbeater.cc:503] Master 127.0.97.62:42529 was elected leader, sending a full tablet report...
W20260619 04:34:02.705330 7108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:34:02.705442 7108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:02.705528 7108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:02.705607 7108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:34:02.705665 7108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:34:02.715857 7108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42529,127.0.97.61:39847,127.0.97.60:39307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:02.717193 7108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:02.718411 7108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:02.729843 7197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:02.729871 7196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:02.729810 7199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:02.732401 7108 server_base.cc:1034] running on GCE node
I20260619 04:34:02.733170 7108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:02.734546 7108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:02.735764 7108 hybrid_clock.cc:648] HybridClock initialized: now 1781843642735712 us; error 47 us; skew 500 ppm
I20260619 04:34:02.736085 7108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:02 dist-test-slave-n3tq krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843642, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:02.738982 7108 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260619 04:34:02.740792 7108 webserver.cc:460] Webserver started at http://127.0.97.60:40149/ using document root <none> and password file <none>
I20260619 04:34:02.741485 7108 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:02.741604 7108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:02.741875 7108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:02.744827 7108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/data/instance:
uuid: "974acf9a4cd54e84b8fe000bf11137a8"
format_stamp: "Formatted at 2026-06-19 04:34:02 on dist-test-slave-n3tq"
server_key: "7c78510752591bab09ce742d39a30077"
server_key_iv: "cdaa0a433f3b3514029cb6e7e9c5ec47"
server_key_version: "encryptionkey@0"
I20260619 04:34:02.745553 7108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/wal/instance:
uuid: "974acf9a4cd54e84b8fe000bf11137a8"
format_stamp: "Formatted at 2026-06-19 04:34:02 on dist-test-slave-n3tq"
server_key: "7c78510752591bab09ce742d39a30077"
server_key_iv: "cdaa0a433f3b3514029cb6e7e9c5ec47"
server_key_version: "encryptionkey@0"
I20260619 04:34:02.750416 7108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:34:02.753708 7206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:02.755039 7108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:34:02.755191 7108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/wal
uuid: "974acf9a4cd54e84b8fe000bf11137a8"
format_stamp: "Formatted at 2026-06-19 04:34:02 on dist-test-slave-n3tq"
server_key: "7c78510752591bab09ce742d39a30077"
server_key_iv: "cdaa0a433f3b3514029cb6e7e9c5ec47"
server_key_version: "encryptionkey@0"
I20260619 04:34:02.755358 7108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:02.777503 7108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:02.782033 7108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:02.782279 7108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:02.800138 7108 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:39307
I20260619 04:34:02.800140 7257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:39307 every 8 connection(s)
I20260619 04:34:02.801576 7108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/data/info.pb
I20260619 04:34:02.805639 7258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:02.807435 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 7108
I20260619 04:34:02.814929 7258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:34:02 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 19 04:34:02 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843642, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 04:34:02.829517 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.809560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41542 (local address 127.0.97.60:39307)
0619 04:34:02.810381 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:02.810393 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:02.810435 (+ 42us) server_negotiation.cc:407] Connection header received
0619 04:34:02.811979 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:02.812002 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:02.812500 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:02.812840 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:02.814550 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.815890 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.817431 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.818451 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.818630 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.822545 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:02.822552 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:02.822564 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:02.822598 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:02.825387 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.825984 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.825991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.825997 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.826139 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.826686 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.826689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.826691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.827256 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:02.827513 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:02.827935 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 04:34:02.828504 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":375,"threads_started":1}
I20260619 04:34:02.830319 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.817505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.0.1:57992)
0619 04:34:02.818183 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:02.818221 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:02.818360 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:02.818692 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:02.818699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:02.819392 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:02.820010 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.820018 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.821001 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.821006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.822012 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.822022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.822655 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.822661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.822814 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.823483 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:02.823498 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:02.825618 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:02.828104 (+ 2486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.828114 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.828121 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.828558 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.828876 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.828881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.828885 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.829016 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.829470 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:02.829481 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:02.829633 (+ 152us) client_negotiation.cc:769] Sending connection context
0619 04:34:02.829825 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 04:34:02.830050 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":348,"threads_started":1}
I20260619 04:34:02.830737 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.817576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57992 (local address 127.0.97.62:42529)
0619 04:34:02.817716 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:02.817720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:02.818309 (+ 589us) server_negotiation.cc:407] Connection header received
0619 04:34:02.818475 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:02.818479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:02.818537 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:02.818620 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:02.820149 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.820888 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.822132 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.822492 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.822591 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.825782 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:02.825786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:02.825789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:02.825818 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:02.827994 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.828661 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.828665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.828667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.828779 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.829115 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.829120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.829123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.829355 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:02.829482 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:02.829938 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 04:34:02.830157 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 04:34:02.832037 7258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:34:02 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843642, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 04:34:02.843748 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.832814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39847 (local address 127.0.0.1:43392)
0619 04:34:02.832983 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:02.833001 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:02.833080 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:02.833513 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:02.833516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:02.833794 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:02.833918 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.833925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.835160 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.835163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.836052 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.836062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.836795 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.836799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.836928 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.837431 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:02.837446 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:02.839210 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:02.841813 (+ 2603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.841817 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.841819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.842170 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.842509 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.842513 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.842515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.842626 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.843030 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:02.843033 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:02.843222 (+ 189us) client_negotiation.cc:769] Sending connection context
0619 04:34:02.843339 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 04:34:02.843536 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 04:34:02.843842 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.832823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43392 (local address 127.0.97.61:39847)
0619 04:34:02.833286 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:02.833290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:02.833305 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:34:02.833350 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:02.833353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:02.833406 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:02.833495 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:02.834066 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.835059 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.836189 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.836672 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.836767 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.839376 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:02.839381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:02.839384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:02.839427 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:02.841705 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.842269 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.842273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.842276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.842400 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.842698 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.842701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.842703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.842938 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:02.843065 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:02.843417 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 04:34:02.843586 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":255,"threads_started":1}
I20260619 04:34:02.844841 7258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:34:02 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843642, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 04:34:02.856258 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.845528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39307 (local address 127.0.0.1:41550)
0619 04:34:02.845693 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:02.845709 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:02.845790 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:02.846014 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:02.846017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:02.846300 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:02.846438 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.846445 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.847315 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.847318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.848254 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.848265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.848945 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.848950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.849097 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.849540 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:02.849554 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:02.851297 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:02.854406 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.854410 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.854412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.854770 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.855039 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.855042 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.855045 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.855170 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.855577 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:02.855580 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:02.855687 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 04:34:02.855788 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 04:34:02.855962 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 04:34:02.856307 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.845505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41550 (local address 127.0.97.60:39307)
0619 04:34:02.845664 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:02.845669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:02.845760 (+ 91us) server_negotiation.cc:407] Connection header received
0619 04:34:02.845863 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:02.845866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:02.845915 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:02.845983 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:02.846560 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.847226 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.848359 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.848809 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.848902 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.851441 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:02.851445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:02.851448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:02.851475 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:02.854276 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.854859 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.854862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.854865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.854966 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.855248 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.855251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.855253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.855490 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:02.855590 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:02.855848 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 04:34:02.856052 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":50}
I20260619 04:34:02.860945 7258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8: Bootstrap starting.
I20260619 04:34:02.863893 7258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:02.864907 7258 log.cc:824] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8: Log is configured to *not* fsync() on all Append() calls
I20260619 04:34:02.867689 7258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8: No bootstrap required, opened a new log
I20260619 04:34:02.873112 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.867021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43402 (local address 127.0.97.61:39847)
0619 04:34:02.867179 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:02.867185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:02.867893 (+ 708us) server_negotiation.cc:407] Connection header received
0619 04:34:02.868075 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:02.868079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:02.868167 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:02.868255 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:02.869111 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.869838 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.871351 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.872223 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.872348 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.872581 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:02.872746 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 04:34:02.872874 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 04:34:02.873150 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.866933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39847 (local address 127.0.0.1:43402)
0619 04:34:02.867377 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:02.867394 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:02.867945 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:02.868349 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:02.868354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:02.868383 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:02.868969 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.868980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.869959 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.869964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.871209 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.871221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.872379 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.872385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.872578 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.872634 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 04:34:02.872715 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 04:34:02.872845 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":255,"threads_started":1}
I20260619 04:34:02.874064 7152 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260619 04:34:02.874534 7180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71decb2899a44f1fb9b71a06ff359c0f. Latest consensus state: current_term: 1 leader_uuid: "71decb2899a44f1fb9b71a06ff359c0f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: true } } }
I20260619 04:34:02.874795 7180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [sys.catalog]: This master's current role is: LEARNER
I20260619 04:34:02.874964 7258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } }
I20260619 04:34:02.875263 7258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:02.875358 7258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974acf9a4cd54e84b8fe000bf11137a8, State: Initialized, Role: FOLLOWER
I20260619 04:34:02.875895 7258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } }
I20260619 04:34:02.876703 7267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } } }
I20260619 04:34:02.876967 7267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:34:02.877640 7258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:34:02.884598 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 6606
W20260619 04:34:02.887132 7278 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 04:34:02.887248 7278 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 04:34:02.890889 7108 master_runner.cc:428] Detected that this master 974acf9a4cd54e84b8fe000bf11137a8 is joining an existing cluster
I20260619 04:34:02.890983 7108 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:39307
I20260619 04:34:02.898010 7267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
Jun 19 04:34:02 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260619 04:34:02.898312 7267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } }
I20260619 04:34:02.899839 7267 leader_election.cc:288] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71decb2899a44f1fb9b71a06ff359c0f (127.0.97.62:42529), 7a74506f385f403d9104e5dcdb2c851a (127.0.97.61:39847)
I20260619 04:34:02.900858 6555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "974acf9a4cd54e84b8fe000bf11137a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71decb2899a44f1fb9b71a06ff359c0f" is_pre_election: true
I20260619 04:34:02.901991 7152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "974acf9a4cd54e84b8fe000bf11137a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a74506f385f403d9104e5dcdb2c851a" is_pre_election: true
I20260619 04:34:02.903111 7210 leader_election.cc:302] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974acf9a4cd54e84b8fe000bf11137a8; no voters: 71decb2899a44f1fb9b71a06ff359c0f, 7a74506f385f403d9104e5dcdb2c851a
I20260619 04:34:02.903760 7267 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 04:34:02.908260 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.885762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52766 (local address 127.0.97.1:38759)
0619 04:34:02.887746 (+ 1984us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:02.887755 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:02.887783 (+ 28us) server_negotiation.cc:407] Connection header received
0619 04:34:02.887846 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:02.887853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:02.888103 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:02.888263 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:02.889555 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.891098 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.892951 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.893837 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.893965 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.899908 (+ 5943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:02.899925 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:02.899945 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:02.899990 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:02.904750 (+ 4760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.905566 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.905575 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.905578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.905727 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.906163 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.906169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.906173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.906638 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:02.906920 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:02.907530 (+ 610us) server_negotiation.cc:299] Negotiation successful
0619 04:34:02.907832 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1823,"thread_start_us":1656,"threads_started":1}
I20260619 04:34:02.910706 6729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:02.938192 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.907240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41554 (local address 127.0.97.60:39307)
0619 04:34:02.907386 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:02.907391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:02.916666 (+ 9275us) server_negotiation.cc:407] Connection header received
0619 04:34:02.918874 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:02.918880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:02.918938 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:02.918994 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:02.923666 (+ 4672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.924786 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.926567 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.927175 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.927258 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.930709 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:02.930714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:02.930718 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:02.930747 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:02.934333 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.935101 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.935106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.935109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.935246 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.936516 (+ 1270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.936521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.936524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.936978 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:02.937132 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:02.937620 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 04:34:02.937836 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 04:34:02.939162 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.906899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39307 (local address 127.0.0.1:41554)
0619 04:34:02.916574 (+ 9675us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:02.916599 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:02.918764 (+ 2165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:02.919147 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:02.919152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:02.919825 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:02.920143 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.920155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.924993 (+ 4838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.924997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.926450 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.926460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.927509 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.927515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.927676 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.928238 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:02.928281 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:02.930574 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:02.934473 (+ 3899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.934478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.934482 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.934992 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.935365 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.935369 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.935372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.935513 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.937197 (+ 1684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:02.937201 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:02.937352 (+ 151us) client_negotiation.cc:769] Sending connection context
0619 04:34:02.938449 (+ 1097us) client_negotiation.cc:240] Negotiation successful
0619 04:34:02.938723 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9529,"mutex_wait_us":1,"spinlock_wait_cycles":36224,"thread_start_us":13557,"threads_started":1}
I20260619 04:34:02.942051 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.907649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43408 (local address 127.0.97.61:39847)
0619 04:34:02.907833 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:02.907839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:02.920060 (+ 12221us) server_negotiation.cc:407] Connection header received
0619 04:34:02.920518 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:02.920524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:02.920599 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:02.920666 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:02.922264 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.923288 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.927169 (+ 3881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.927782 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.927879 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.931587 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:02.931594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:02.931599 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:02.931641 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:02.934743 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.935411 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.935417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.935420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.935553 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.935963 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.935968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.935971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.936296 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:02.939186 (+ 2890us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:02.941490 (+ 2304us) server_negotiation.cc:299] Negotiation successful
0619 04:34:02.941729 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 04:34:02.942231 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.907194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58006 (local address 127.0.97.62:42529)
0619 04:34:02.907339 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:02.907346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:02.915311 (+ 7965us) server_negotiation.cc:407] Connection header received
0619 04:34:02.917049 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:02.917057 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:02.917146 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:02.917218 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:02.920305 (+ 3087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.921909 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.925259 (+ 3350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.925879 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.925975 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.935814 (+ 9839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:02.935820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:02.935825 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:02.935863 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:02.938989 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.940117 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.940122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.940125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.940265 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.940737 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.940741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.940744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.941058 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:02.941192 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:02.941716 (+ 524us) server_negotiation.cc:299] Negotiation successful
0619 04:34:02.941916 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 04:34:02.942973 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.908159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39847 (local address 127.0.0.1:43408)
0619 04:34:02.919947 (+ 11788us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:02.919963 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:02.920340 (+ 377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:02.920847 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:02.920850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:02.921796 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:02.922105 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.922114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.923467 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.923473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.926197 (+ 2724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.926213 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.928176 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.928187 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.928375 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.929895 (+ 1520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:02.929920 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:02.931392 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:02.934880 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.934885 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.934888 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.935298 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.935686 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.935690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.935694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.935840 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.939263 (+ 3423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:02.939267 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:02.941203 (+ 1936us) client_negotiation.cc:769] Sending connection context
0619 04:34:02.942295 (+ 1092us) client_negotiation.cc:240] Negotiation successful
0619 04:34:02.942669 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11665,"mutex_wait_us":1099,"thread_start_us":8516,"threads_started":1}
I20260619 04:34:02.943740 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.906743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.0.1:58006)
0619 04:34:02.915201 (+ 8458us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:02.915222 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:02.915999 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:02.917433 (+ 1434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:02.917439 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:02.918175 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:02.918588 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.918601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.922093 (+ 3492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.922097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.925090 (+ 2993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.925106 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.926430 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.926435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.926601 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.928949 (+ 2348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:02.928971 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:02.930628 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:02.939633 (+ 9005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.939637 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.939640 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.939986 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.940404 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.940409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.940412 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.940615 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.941263 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:02.941267 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:02.941459 (+ 192us) client_negotiation.cc:769] Sending connection context
0619 04:34:02.942373 (+ 914us) client_negotiation.cc:240] Negotiation successful
0619 04:34:02.943346 (+ 973us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8269,"mutex_wait_us":2142,"thread_start_us":13347,"threads_started":1}
Jun 19 04:34:02 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843640, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 04:34:02.959308 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.948290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39307 (local address 127.0.0.1:41564)
0619 04:34:02.948569 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:02.948586 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:02.948691 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:02.949037 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:02.949042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:02.949385 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:02.949510 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.949518 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.950459 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.950464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.951289 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:02.951299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.951967 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.951973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:02.952149 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.952660 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:02.952676 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:02.954490 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:02.957304 (+ 2814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.957309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.957313 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.957674 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.957986 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:02.957991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:02.957995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:02.958112 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:02.958596 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:02.958601 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:02.958749 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 04:34:02.958871 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 04:34:02.959074 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260619 04:34:02.960131 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:02.948357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41564 (local address 127.0.97.60:39307)
0619 04:34:02.948561 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:02.948566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:02.948632 (+ 66us) server_negotiation.cc:407] Connection header received
0619 04:34:02.948832 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:02.948837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:02.948916 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:02.948994 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:02.949641 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.950326 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.951413 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:02.951845 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:02.951945 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:02.954654 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:02.954659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:02.954665 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:02.954693 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:02.957184 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.957776 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.957781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.957783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.957890 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:02.958214 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:02.958219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:02.958223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:02.958479 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:02.958613 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:02.959642 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0619 04:34:02.959839 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 04:34:02.961325 6545 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:39307: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 39307
}
attrs {
promote: true
}
}
dest_uuid: "71decb2899a44f1fb9b71a06ff359c0f"
cas_config_opid_index: 5
I20260619 04:34:02.962141 6545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: true } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: true } }
I20260619 04:34:02.963670 7265 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 974acf9a4cd54e84b8fe000bf11137a8 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: true } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: true } } }
I20260619 04:34:02.964425 7152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 1 LEARNER]: Refusing update from remote peer 71decb2899a44f1fb9b71a06ff359c0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 04:34:02.965837 7233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:02.969846 7233 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 1 FOLLOWER]: Refusing update from remote peer 71decb2899a44f1fb9b71a06ff359c0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 04:34:02.976892 7265 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:39307
I20260619 04:34:02.977068 7290 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 04:34:02.980479 7152 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 974acf9a4cd54e84b8fe000bf11137a8 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: true } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: true } } }
I20260619 04:34:02.981665 7108 master.cc:537] Master@127.0.97.60:39307 shutting down...
I20260619 04:34:02.981634 7289 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.001s
I20260619 04:34:02.985612 7180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71decb2899a44f1fb9b71a06ff359c0f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: true } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: true } } }
I20260619 04:34:02.985910 7180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [sys.catalog]: This master's current role is: LEARNER
I20260619 04:34:02.994977 7291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "71decb2899a44f1fb9b71a06ff359c0f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: true } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: true } } }
I20260619 04:34:02.995572 7291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [sys.catalog]: This master's current role is: LEADER
W20260619 04:34:02.997280 6530 consensus_peers.cc:487] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f -> Peer 974acf9a4cd54e84b8fe000bf11137a8 (127.0.97.60:39307): Couldn't send request to peer 974acf9a4cd54e84b8fe000bf11137a8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 04:34:02.997689 7290 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f: attempting to promote NON_VOTER 7a74506f385f403d9104e5dcdb2c851a to VOTER
I20260619 04:34:03.003885 7290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: true } }
I20260619 04:34:03.006565 7152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 1 LEARNER]: Refusing update from remote peer 71decb2899a44f1fb9b71a06ff359c0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260619 04:34:03.006628 6530 consensus_peers.cc:487] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f -> Peer 974acf9a4cd54e84b8fe000bf11137a8 (127.0.97.60:39307): Couldn't send request to peer 974acf9a4cd54e84b8fe000bf11137a8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 04:34:03.007145 7290 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 04:34:03.009231 7291 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7a74506f385f403d9104e5dcdb2c851a (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: true } } }
I20260619 04:34:03.009989 7152 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7a74506f385f403d9104e5dcdb2c851a (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: true } } }
I20260619 04:34:03.012741 7180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71decb2899a44f1fb9b71a06ff359c0f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: true } } }
I20260619 04:34:03.013021 7180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:34:03.015763 7265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "71decb2899a44f1fb9b71a06ff359c0f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: true } } }
I20260619 04:34:03.015965 7265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [sys.catalog]: This master's current role is: LEADER
I20260619 04:34:03.016309 7291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71decb2899a44f1fb9b71a06ff359c0f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: true } } }
I20260619 04:34:03.016525 7291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [sys.catalog]: This master's current role is: LEADER
I20260619 04:34:03.022392 7108 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:34:03.022779 7108 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:34:03.022961 7108 tablet_replica.cc:330] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8: stopping tablet replica
I20260619 04:34:03.028743 7108 master.cc:552] Master@127.0.97.60:39307 shutdown complete.
I20260619 04:34:03.028820 7108 master_runner.cc:305] Clearing existing system tablet
I20260619 04:34:03.029160 7108 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:03.029242 7108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:03.031276 7108 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 04:34:03.032564 7297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:03.032968 7108 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:03.033253 7108 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 04:34:03.033349 7108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/wal
uuid: "974acf9a4cd54e84b8fe000bf11137a8"
format_stamp: "Formatted at 2026-06-19 04:34:02 on dist-test-slave-n3tq"
server_key: "7c78510752591bab09ce742d39a30077"
server_key_iv: "cdaa0a433f3b3514029cb6e7e9c5ec47"
server_key_version: "encryptionkey@0"
I20260619 04:34:03.036679 7108 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:03.041599 7108 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 04:34:03.041738 7108 log.cc:1196] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 04:34:03.042143 7108 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8: Deleting consensus metadata
I20260619 04:34:03.042357 7108 master_runner.cc:315] Copying system tablet from 127.0.97.62:42529
I20260619 04:34:03.044617 7108 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:42529
I20260619 04:34:03.053167 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 6606
I20260619 04:34:03.063982 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:03.045320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58010 (local address 127.0.97.62:42529)
0619 04:34:03.045566 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:03.045574 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:03.049016 (+ 3442us) server_negotiation.cc:407] Connection header received
0619 04:34:03.049200 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:03.049205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:03.049267 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:03.049347 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:03.050215 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:03.051319 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:03.053036 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:03.053601 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:03.053699 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:03.059023 (+ 5324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:03.059028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:03.059032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:03.059062 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:03.061502 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:03.062126 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:03.062130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:03.062132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:03.062248 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:03.062607 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:03.062611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:03.062613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:03.062872 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:03.063011 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:03.063389 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 04:34:03.063622 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260619 04:34:03.064002 7303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:03.045275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.0.1:58010)
0619 04:34:03.048890 (+ 3615us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:03.048910 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:03.049068 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:03.049389 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:03.049394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:03.049820 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:03.050037 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:03.050067 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:03.051446 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:03.051451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:03.052867 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:03.052880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:03.054695 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:03.054699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:03.054835 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:03.058073 (+ 3238us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:03.058102 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:03.058832 (+ 730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:03.061647 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:03.061652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:03.061656 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:03.062003 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:03.062350 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:03.062355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:03.062359 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:03.062493 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:03.062980 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:03.062985 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:03.063161 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 04:34:03.063269 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 04:34:03.063572 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3491,"thread_start_us":3319,"threads_started":1}
I20260619 04:34:03.064621 6565 tablet_copy_service.cc:140] P 71decb2899a44f1fb9b71a06ff359c0f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 974acf9a4cd54e84b8fe000bf11137a8 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:58010)
I20260619 04:34:03.064766 6565 tablet_copy_service.cc:161] P 71decb2899a44f1fb9b71a06ff359c0f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 974acf9a4cd54e84b8fe000bf11137a8 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:58010: session id = 974acf9a4cd54e84b8fe000bf11137a8-00000000000000000000000000000000
I20260619 04:34:03.066926 6565 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 04:34:03.068414 7108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 04:34:03.072917 7108 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8: tablet copy: Starting download of 0 data blocks...
I20260619 04:34:03.073186 7108 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8: tablet copy: Starting download of 1 WAL segments...
I20260619 04:34:03.073848 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:38759
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42529
--webserver_port=34785
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:42529,127.0.97.61:39847
--tserver_master_addrs=127.0.97.62:42529,127.0.97.61:39847,127.0.97.60:39307 with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:34:03.075121 7108 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 04:34:03.078156 6565 tablet_copy_service.cc:342] P 71decb2899a44f1fb9b71a06ff359c0f: Request end of tablet copy session 974acf9a4cd54e84b8fe000bf11137a8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:58010
I20260619 04:34:03.078274 6565 tablet_copy_service.cc:433] P 71decb2899a44f1fb9b71a06ff359c0f: ending tablet copy session 974acf9a4cd54e84b8fe000bf11137a8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 974acf9a4cd54e84b8fe000bf11137a8
I20260619 04:34:03.088315 7108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:03.090864 7305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:03.100761 7306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:03.104522 7308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:03.104984 7108 server_base.cc:1034] running on GCE node
I20260619 04:34:03.105283 7108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:03.105880 7108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:03.107052 7108 hybrid_clock.cc:648] HybridClock initialized: now 1781843643107010 us; error 37 us; skew 500 ppm
I20260619 04:34:03.107261 7108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:03 dist-test-slave-n3tq krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843643, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:03.109035 7108 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260619 04:34:03.110280 7108 webserver.cc:460] Webserver started at http://127.0.97.60:38631/ using document root <none> and password file <none>
I20260619 04:34:03.110544 7108 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:03.110643 7108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:03.113116 7108 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260619 04:34:03.116866 7315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:03.117228 7108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:34:03.117349 7108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/wal
uuid: "974acf9a4cd54e84b8fe000bf11137a8"
format_stamp: "Formatted at 2026-06-19 04:34:02 on dist-test-slave-n3tq"
server_key: "7c78510752591bab09ce742d39a30077"
server_key_iv: "cdaa0a433f3b3514029cb6e7e9c5ec47"
server_key_version: "encryptionkey@0"
I20260619 04:34:03.117511 7108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:03.127810 7108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:03.130960 7108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:03.151800 7108 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:39307
I20260619 04:34:03.151831 7366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:39307 every 8 connection(s)
I20260619 04:34:03.152701 7108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/master-2/data/info.pb
I20260619 04:34:03.158450 7367 sys_catalog.cc:263] Verifying existing consensus state
I20260619 04:34:03.161624 7367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8: Bootstrap starting.
I20260619 04:34:03.187202 7367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 04:34:03.187705 7367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8: Bootstrap complete.
I20260619 04:34:03.188809 7367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: true } }
I20260619 04:34:03.189102 7367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 974acf9a4cd54e84b8fe000bf11137a8, State: Initialized, Role: LEARNER
I20260619 04:34:03.189370 7367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: true } }
I20260619 04:34:03.190152 7370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: true } } }
I20260619 04:34:03.190454 7367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:34:03.190657 7370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [sys.catalog]: This master's current role is: LEARNER
I20260619 04:34:03.211661 7381 catalog_manager.cc:1259] Loaded cluster ID: 7704a7aa6f7e4cddaf287d60a6d4102c
I20260619 04:34:03.211769 7381 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8: loading cluster ID for follower catalog manager: success
I20260619 04:34:03.215294 7381 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8: acquiring CA information for follower catalog manager: success
I20260619 04:34:03.217593 7381 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 04:34:03.406287 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:03.400705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39307 (local address 127.0.0.1:41572)
0619 04:34:03.400922 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:03.400941 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:03.401062 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:03.401784 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:03.401789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:03.401816 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:03.401994 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:03.402002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:03.403235 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:03.403241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:03.404351 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:03.404364 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:03.405486 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:03.405491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:03.405717 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:03.405752 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 04:34:03.405840 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 04:34:03.405983 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 04:34:03.406320 7382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:03.400911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41572 (local address 127.0.97.60:39307)
0619 04:34:03.401458 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:03.401463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:03.401479 (+ 16us) server_negotiation.cc:407] Connection header received
0619 04:34:03.401535 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:03.401539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:03.401611 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:03.401715 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:03.402141 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:03.403077 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:03.404533 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:03.405324 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:03.405439 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:03.405681 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:03.405872 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 04:34:03.405995 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":284,"threads_started":1}
I20260619 04:34:03.407226 7342 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 04:34:03.407752 7370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71decb2899a44f1fb9b71a06ff359c0f. Latest consensus state: current_term: 1 leader_uuid: "71decb2899a44f1fb9b71a06ff359c0f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: true } } }
I20260619 04:34:03.408041 7370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [sys.catalog]: This master's current role is: LEARNER
W20260619 04:34:03.432111 7304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:03.432380 7304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:03.432539 7304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:03.439687 7304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:03.439793 7304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:03.439944 7304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:34:03.452921 7304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:38759
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34785
--tserver_master_addrs=127.0.97.62:42529,127.0.97.61:39847,127.0.97.60:39307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:03.454461 7304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:03.456053 7304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:03.472740 7387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:03.472795 7388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:03.472775 7390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:03.473469 7304 server_base.cc:1034] running on GCE node
I20260619 04:34:03.474061 7304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:03.474994 7304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:03.476187 7304 hybrid_clock.cc:648] HybridClock initialized: now 1781843643476125 us; error 51 us; skew 500 ppm
I20260619 04:34:03.476518 7304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:03 dist-test-slave-n3tq krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843643, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:03.479516 7304 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 04:34:03.480932 7304 webserver.cc:460] Webserver started at http://127.0.97.1:34785/ using document root <none> and password file <none>
I20260619 04:34:03.481578 7304 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:03.481714 7304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:03.487290 7304 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 04:34:03.489938 7397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:03.491170 7304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:34:03.491307 7304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "6af077b208f048a2bc017df0886b3e70"
format_stamp: "Formatted at 2026-06-19 04:34:00 on dist-test-slave-n3tq"
server_key: "71ee397c4b6c8c23b0f6cd6a66bea4a4"
server_key_iv: "d5b075179135f7b28c6b6374248e6b90"
server_key_version: "encryptionkey@0"
I20260619 04:34:03.492245 7304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:03.507781 7304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:03.512292 7304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:03.512570 7304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:03.513640 7304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:34:03.515021 7304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:03.515113 7304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:03.515214 7304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:03.515286 7304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:03.547818 7304 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38759
I20260619 04:34:03.547878 7509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38759 every 8 connection(s)
I20260619 04:34:03.549544 7304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:34:03.549953 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 7304
I20260619 04:34:03.550213 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 6740
Jun 19 04:34:03 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843643, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 19 04:34:03 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843643, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 04:34:03.575713 7382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:03.552492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45961 (local address 127.0.97.60:39307)
0619 04:34:03.552677 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:03.552683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:03.554022 (+ 1339us) server_negotiation.cc:407] Connection header received
0619 04:34:03.557446 (+ 3424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:03.557453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:03.557551 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:03.557628 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:03.559969 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:03.561103 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:03.563539 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:03.564178 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:03.564268 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:03.569203 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:03.569208 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:03.569211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:03.569244 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:03.572256 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:03.573172 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:03.573178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:03.573181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:03.573311 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:03.573814 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:03.573819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:03.573823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:03.574151 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:03.574267 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:03.575159 (+ 892us) server_negotiation.cc:299] Negotiation successful
0619 04:34:03.575377 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:34:03.576977 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:03.552344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35207 (local address 127.0.97.62:42529)
0619 04:34:03.552556 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:03.552563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:03.554365 (+ 1802us) server_negotiation.cc:407] Connection header received
0619 04:34:03.557466 (+ 3101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:03.557473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:03.557567 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:03.557639 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:03.560375 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:03.561512 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:03.563439 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:03.564114 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:03.564239 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:03.570395 (+ 6156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:03.570402 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:03.570407 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:03.570449 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:03.573609 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:03.574259 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:03.574265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:03.574269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:03.574409 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:03.575473 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:03.575478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:03.575481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:03.575817 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:03.575951 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:03.576319 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 04:34:03.576588 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 04:34:03.578431 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:03.553007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39307 (local address 127.0.97.1:45961)
0619 04:34:03.553853 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:03.553909 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:03.557108 (+ 3199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:03.558160 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:03.558174 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:03.559120 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:03.559797 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:03.559815 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:03.561290 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:03.561296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:03.563368 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:03.563382 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:03.564642 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:03.564649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:03.565005 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:03.565816 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:03.565852 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:03.568967 (+ 3115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:03.572410 (+ 3443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:03.572422 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:03.572441 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:03.573043 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:03.573445 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:03.573450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:03.573454 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:03.573698 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:03.574351 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:03.574359 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:03.574759 (+ 400us) client_negotiation.cc:769] Sending connection context
0619 04:34:03.576209 (+ 1450us) client_negotiation.cc:240] Negotiation successful
0619 04:34:03.576795 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":101,"thread_start_us":1614,"threads_started":1}
I20260619 04:34:03.578459 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:03.553585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.97.1:35207)
0619 04:34:03.554260 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:03.554278 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:03.557122 (+ 2844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:03.558161 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:03.558174 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:03.560028 (+ 1854us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:03.560232 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:03.560239 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:03.562134 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:03.562138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:03.563302 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:03.563314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:03.564654 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:03.564661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:03.565412 (+ 751us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:03.565912 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:03.565940 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:03.570197 (+ 4257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:03.573700 (+ 3503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:03.573703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:03.573707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:03.574138 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:03.575218 (+ 1080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:03.575222 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:03.575226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:03.575353 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:03.575937 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:03.575941 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:03.576086 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 04:34:03.576213 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 04:34:03.576908 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":621,"thread_start_us":338,"threads_started":1}
Jun 19 04:34:03 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843643, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 19 04:34:03 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260619 04:34:03.580073 7511 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42529
I20260619 04:34:03.580348 7510 heartbeater.cc:347] Connected to a master server at 127.0.97.60:39307
I20260619 04:34:03.580397 7511 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:03.580513 7510 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:03.581389 7511 heartbeater.cc:511] Master 127.0.97.62:42529 requested a full tablet report, sending...
I20260619 04:34:03.581409 7510 heartbeater.cc:511] Master 127.0.97.60:39307 requested a full tablet report, sending...
I20260619 04:34:03.583460 6545 ts_manager.cc:194] Re-registered known tserver with Master: 6af077b208f048a2bc017df0886b3e70 (127.0.97.1:38759)
I20260619 04:34:03.583791 7332 ts_manager.cc:194] Registered new tserver with Master: 6af077b208f048a2bc017df0886b3e70 (127.0.97.1:38759)
I20260619 04:34:03.584406 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35207
I20260619 04:34:03.588106 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:03.557453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39847 (local address 127.0.97.1:35005)
0619 04:34:03.566428 (+ 8975us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:03.566448 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:03.566604 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:03.567154 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:03.567159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:03.568234 (+ 1075us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:03.568639 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:03.568653 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:03.572122 (+ 3469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:03.572128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:03.573562 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:03.573573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:03.575542 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:03.575548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:03.575703 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:03.577283 (+ 1580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:03.577305 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:03.580840 (+ 3535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:03.585074 (+ 4234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:03.585079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:03.585083 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:03.585529 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:03.585931 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:03.585935 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:03.585939 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:03.586102 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:03.587177 (+ 1075us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:03.587182 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:03.587322 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 04:34:03.587448 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 04:34:03.587741 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8826,"thread_start_us":8659,"threads_started":1}
I20260619 04:34:03.588145 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:03.557932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35005 (local address 127.0.97.61:39847)
0619 04:34:03.565365 (+ 7433us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:03.565371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:03.566552 (+ 1181us) server_negotiation.cc:407] Connection header received
0619 04:34:03.566721 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:03.566727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:03.566813 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:03.566918 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:03.568845 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:03.570921 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:03.574285 (+ 3364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:03.574909 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:03.575027 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:03.581081 (+ 6054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:03.581088 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:03.581093 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:03.581133 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:03.584949 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:03.585664 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:03.585670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:03.585673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:03.585815 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:03.586231 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:03.586235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:03.586238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:03.587067 (+ 829us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:03.587179 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:03.587579 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 04:34:03.587785 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7291,"thread_start_us":7155,"threads_started":1}
I20260619 04:34:03.588693 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:03.551470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57024 (local address 127.0.97.2:37011)
0619 04:34:03.567144 (+ 15674us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:03.567150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:03.567175 (+ 25us) server_negotiation.cc:407] Connection header received
0619 04:34:03.567225 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:03.567231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:03.567502 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:03.569656 (+ 2154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:03.569898 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:03.571890 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:03.573753 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:03.574354 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:03.574947 (+ 593us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:03.581079 (+ 6132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:03.581093 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:03.581109 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:03.581151 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:03.585903 (+ 4752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:03.586513 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:03.586521 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:03.586524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:03.586636 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:03.586971 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:03.586976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:03.586980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:03.587336 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:03.587615 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:03.587902 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 04:34:03.588151 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15524,"thread_start_us":15381,"threads_started":1}
I20260619 04:34:03.589773 6863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:03.589946 7512 heartbeater.cc:347] Connected to a master server at 127.0.97.61:39847
I20260619 04:34:03.590077 7512 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:03.590482 7512 heartbeater.cc:511] Master 127.0.97.61:39847 requested a full tablet report, sending...
I20260619 04:34:03.592136 7142 ts_manager.cc:194] Registered new tserver with Master: 6af077b208f048a2bc017df0886b3e70 (127.0.97.1:38759)
I20260619 04:34:03.671305 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 6740
I20260619 04:34:03.684080 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37011
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42529
--webserver_port=33225
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:42529,127.0.97.61:39847
--tserver_master_addrs=127.0.97.62:42529,127.0.97.61:39847,127.0.97.60:39307 with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:34:03.870339 7290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71decb2899a44f1fb9b71a06ff359c0f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: true } } }
I20260619 04:34:03.870669 7290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [sys.catalog]: This master's current role is: LEADER
I20260619 04:34:03.872113 7522 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f: attempting to promote NON_VOTER 974acf9a4cd54e84b8fe000bf11137a8 to VOTER
I20260619 04:34:03.872680 7522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: false } }
I20260619 04:34:03.874445 7152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 1 FOLLOWER]: Refusing update from remote peer 71decb2899a44f1fb9b71a06ff359c0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 04:34:03.874440 7342 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 1 LEARNER]: Refusing update from remote peer 71decb2899a44f1fb9b71a06ff359c0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 04:34:03.874984 7523 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 04:34:03.875391 7522 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 04:34:03.877920 7290 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 974acf9a4cd54e84b8fe000bf11137a8 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: false } } }
I20260619 04:34:03.878472 7152 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 974acf9a4cd54e84b8fe000bf11137a8 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: false } } }
I20260619 04:34:03.880765 7290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "71decb2899a44f1fb9b71a06ff359c0f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: false } } }
I20260619 04:34:03.881088 7290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [sys.catalog]: This master's current role is: LEADER
I20260619 04:34:03.881754 7526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71decb2899a44f1fb9b71a06ff359c0f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: false } } }
I20260619 04:34:03.882220 7526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [sys.catalog]: This master's current role is: LEADER
I20260619 04:34:03.881950 7290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71decb2899a44f1fb9b71a06ff359c0f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: false } } }
I20260619 04:34:03.882498 7290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f [sys.catalog]: This master's current role is: LEADER
I20260619 04:34:03.882776 7527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71decb2899a44f1fb9b71a06ff359c0f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: false } } }
I20260619 04:34:03.883198 7527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:34:03.883467 7342 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 974acf9a4cd54e84b8fe000bf11137a8 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: false } } }
I20260619 04:34:03.886345 7370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71decb2899a44f1fb9b71a06ff359c0f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: false } } }
I20260619 04:34:03.886586 7370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 04:34:04.000780 7521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:04.001062 7521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:04.001190 7521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:04.008419 7521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:04.008570 7521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:04.008728 7521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:34:04.018749 7521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33225
--tserver_master_addrs=127.0.97.62:42529,127.0.97.61:39847,127.0.97.60:39307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:04.020054 7521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:04.021346 7521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:04.033685 7532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:04.033649 7535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:04.033731 7533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:04.034837 7521 server_base.cc:1034] running on GCE node
I20260619 04:34:04.035398 7521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:04.036278 7521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:04.037454 7521 hybrid_clock.cc:648] HybridClock initialized: now 1781843644037391 us; error 57 us; skew 500 ppm
I20260619 04:34:04.037741 7521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:04 dist-test-slave-n3tq krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843644, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:04.040634 7521 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 04:34:04.041965 7521 webserver.cc:460] Webserver started at http://127.0.97.2:33225/ using document root <none> and password file <none>
I20260619 04:34:04.042601 7521 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:04.042722 7521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:04.047782 7521 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:34:04.050379 7542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:04.051605 7521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:34:04.051745 7521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "6f5a5fd7474244c6bfb857386b7e9e0e"
format_stamp: "Formatted at 2026-06-19 04:34:01 on dist-test-slave-n3tq"
server_key: "6f1f84a02f63aced7837a961bbd8e9f4"
server_key_iv: "52eb3ed070e18c6705c95b136f51a94a"
server_key_version: "encryptionkey@0"
I20260619 04:34:04.052757 7521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:04.077478 7521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:04.081748 7521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:04.081990 7521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:04.082942 7521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:34:04.084307 7521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:04.084403 7521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:04.084520 7521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:04.084589 7521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:04.116257 7521 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37011
I20260619 04:34:04.116320 7654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37011 every 8 connection(s)
I20260619 04:34:04.117849 7521 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:34:04.124744 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 7521
I20260619 04:34:04.125005 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 6873
Jun 19 04:34:04 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843644, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 19 04:34:04 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Jun 19 04:34:04 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843644, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 19 04:34:04 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843644, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 04:34:04.175292 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.126436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51966 (local address 127.0.97.3:42411)
0619 04:34:04.152809 (+ 26373us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:04.152816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:04.152839 (+ 23us) server_negotiation.cc:407] Connection header received
0619 04:34:04.152889 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:04.152896 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:04.153143 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:04.154626 (+ 1483us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:04.156990 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.158611 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.160404 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.161226 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.161333 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:04.167709 (+ 6376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:04.167724 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:04.167740 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:04.167778 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:04.171278 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:04.172186 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:04.172195 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:04.172199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:04.172344 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:04.172841 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:04.172846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:04.172849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:04.173264 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:04.173531 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:04.174648 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0619 04:34:04.174901 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26220,"thread_start_us":26084,"threads_started":1}
I20260619 04:34:04.176829 6996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:04.177138 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.121897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39847 (local address 127.0.97.2:36219)
0619 04:34:04.150655 (+ 28758us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:04.150668 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:04.150740 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:04.157516 (+ 6776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:04.157521 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:04.157989 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:04.158269 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:04.158280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.159717 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.159723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:04.161190 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:04.161204 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.162249 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.162256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:04.162539 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:04.163484 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:04.163511 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:04.166269 (+ 2758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:04.169891 (+ 3622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:04.169900 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:04.169918 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:04.171708 (+ 1790us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:04.172350 (+ 642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:04.172355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:04.172359 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:04.172906 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:04.173639 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:04.173647 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:04.174052 (+ 405us) client_negotiation.cc:769] Sending connection context
0619 04:34:04.175211 (+ 1159us) client_negotiation.cc:240] Negotiation successful
0619 04:34:04.175728 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28650,"thread_start_us":28537,"threads_started":1}
I20260619 04:34:04.178151 7659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.120604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36219 (local address 127.0.97.61:39847)
0619 04:34:04.157137 (+ 36533us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:04.157143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:04.157162 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:34:04.157214 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:04.157219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:04.157288 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:04.158408 (+ 1120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:04.158548 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.159570 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.161384 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.162014 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.163209 (+ 1195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:04.166497 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:04.166504 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:04.166508 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:04.166558 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:04.169748 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:04.172071 (+ 2323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:04.172077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:04.172081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:04.172223 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:04.173051 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:04.173056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:04.173059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:04.173403 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:04.173552 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:04.177602 (+ 4050us) server_negotiation.cc:299] Negotiation successful
0619 04:34:04.177822 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36394,"thread_start_us":36751,"threads_started":1}
I20260619 04:34:04.179805 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.121640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58745 (local address 127.0.97.60:39307)
0619 04:34:04.150852 (+ 29212us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:04.150856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:04.150871 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:34:04.150915 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:04.150919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:04.151006 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:04.151095 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:04.152969 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.154647 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.156860 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.165079 (+ 8219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.165189 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:04.173611 (+ 8422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:04.173619 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:04.173624 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:04.173666 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:04.176556 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:04.177476 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:04.177579 (+ 103us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:04.177583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:04.177732 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:04.178162 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:04.178168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:04.178171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:04.178506 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:04.178638 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:04.179160 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 04:34:04.179395 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26709,"thread_start_us":27239,"threads_started":1}
I20260619 04:34:04.180011 7657 heartbeater.cc:347] Connected to a master server at 127.0.97.61:39847
I20260619 04:34:04.180330 7657 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:04.181329 7657 heartbeater.cc:511] Master 127.0.97.61:39847 requested a full tablet report, sending...
I20260619 04:34:04.181466 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.120682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50761 (local address 127.0.97.62:42529)
0619 04:34:04.149945 (+ 29263us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:04.149953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:04.149977 (+ 24us) server_negotiation.cc:407] Connection header received
0619 04:34:04.150315 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:04.150321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:04.150409 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:04.150509 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:04.153216 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.154350 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.170228 (+ 15878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.170692 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.170809 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:04.174656 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:04.174660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:04.174663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:04.174696 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:04.178701 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:04.179412 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:04.179417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:04.179420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:04.179571 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:04.179972 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:04.179976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:04.179979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:04.180305 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:04.180427 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:04.180920 (+ 493us) server_negotiation.cc:299] Negotiation successful
0619 04:34:04.181137 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29132,"thread_start_us":29308,"threads_started":1}
I20260619 04:34:04.182191 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.121342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.97.2:50761)
0619 04:34:04.145276 (+ 23934us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:04.145344 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:04.148143 (+ 2799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:04.150968 (+ 2825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:04.150984 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:04.151866 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:04.153085 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:04.153095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.154491 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.154496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:04.170081 (+ 15585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:04.170094 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.170883 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.170889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:04.171495 (+ 606us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:04.172054 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:04.172074 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:04.174411 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:04.178843 (+ 4432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:04.178848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:04.178852 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:04.179283 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:04.179701 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:04.179705 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:04.179708 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:04.179859 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:04.180518 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:04.180523 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:04.180646 (+ 123us) client_negotiation.cc:769] Sending connection context
0619 04:34:04.181630 (+ 984us) client_negotiation.cc:240] Negotiation successful
0619 04:34:04.181858 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23497,"mutex_wait_us":14014,"thread_start_us":27167,"threads_started":1}
I20260619 04:34:04.182808 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.121693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39307 (local address 127.0.97.2:58745)
0619 04:34:04.149446 (+ 27753us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:04.149469 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:04.150199 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:04.151357 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:04.151362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:04.151987 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:04.152781 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:04.152807 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.154825 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.154833 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:04.156701 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:04.156714 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.169549 (+ 12835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.169556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:04.169738 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:04.170229 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:04.170253 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:04.173401 (+ 3148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:04.176710 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:04.176715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:04.176718 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:04.177327 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:04.177880 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:04.177885 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:04.177889 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:04.178033 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:04.178720 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:04.178725 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:04.178878 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 04:34:04.180266 (+ 1388us) client_negotiation.cc:240] Negotiation successful
0619 04:34:04.180918 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27637,"mutex_wait_us":79,"thread_start_us":27426,"threads_started":1}
I20260619 04:34:04.183228 7142 ts_manager.cc:194] Registered new tserver with Master: 6f5a5fd7474244c6bfb857386b7e9e0e (127.0.97.2:37011)
I20260619 04:34:04.183666 7656 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42529
I20260619 04:34:04.184219 7656 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:04.184729 7656 heartbeater.cc:511] Master 127.0.97.62:42529 requested a full tablet report, sending...
I20260619 04:34:04.184774 7655 heartbeater.cc:347] Connected to a master server at 127.0.97.60:39307
I20260619 04:34:04.184916 7655 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:04.185335 7655 heartbeater.cc:511] Master 127.0.97.60:39307 requested a full tablet report, sending...
I20260619 04:34:04.185629 6545 ts_manager.cc:194] Re-registered known tserver with Master: 6f5a5fd7474244c6bfb857386b7e9e0e (127.0.97.2:37011)
I20260619 04:34:04.186378 7332 ts_manager.cc:194] Registered new tserver with Master: 6f5a5fd7474244c6bfb857386b7e9e0e (127.0.97.2:37011)
I20260619 04:34:04.186683 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50761
I20260619 04:34:04.268654 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 6873
I20260619 04:34:04.281874 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42411
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42529
--webserver_port=46745
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:42529,127.0.97.61:39847
--tserver_master_addrs=127.0.97.62:42529,127.0.97.61:39847,127.0.97.60:39307 with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:34:04.587090 7511 heartbeater.cc:503] Master 127.0.97.62:42529 was elected leader, sending a full tablet report...
W20260619 04:34:04.596292 7668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:04.596634 7668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:04.596740 7668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:04.604027 7668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:04.604130 7668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:04.604274 7668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:34:04.616317 7668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=46745
--tserver_master_addrs=127.0.97.62:42529,127.0.97.61:39847,127.0.97.60:39307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:04.617630 7668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:04.618974 7668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:04.631349 7676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:04.631382 7674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:04.631443 7673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:04.633781 7668 server_base.cc:1034] running on GCE node
I20260619 04:34:04.634438 7668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:04.635456 7668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:04.636644 7668 hybrid_clock.cc:648] HybridClock initialized: now 1781843644636593 us; error 42 us; skew 500 ppm
I20260619 04:34:04.636925 7668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:04 dist-test-slave-n3tq krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843644, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:04.639923 7668 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 04:34:04.641361 7668 webserver.cc:460] Webserver started at http://127.0.97.3:46745/ using document root <none> and password file <none>
I20260619 04:34:04.642047 7668 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:04.642170 7668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:04.647482 7668 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 04:34:04.650295 7683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:04.651635 7668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:34:04.651777 7668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "7ec7bb13dce74266a393d0693e2b6ff5"
format_stamp: "Formatted at 2026-06-19 04:34:01 on dist-test-slave-n3tq"
server_key: "93d446785745a6a1c5ee801eb7ed8752"
server_key_iv: "8c782012d203074d9b09bad71367dfc6"
server_key_version: "encryptionkey@0"
I20260619 04:34:04.652781 7668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:04.667395 7668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:04.671079 7668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:04.671331 7668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:04.672245 7668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:34:04.673651 7668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:04.673735 7668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:04.673831 7668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:04.673902 7668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:04.702940 7668 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42411
I20260619 04:34:04.702980 7795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42411 every 8 connection(s)
I20260619 04:34:04.704362 7668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:34:04.710734 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 7668
Jun 19 04:34:04 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843644, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 19 04:34:04 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843644, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 19 04:34:04 dist-test-slave-n3tq krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843644, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 04:34:04.749897 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.707873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39307 (local address 127.0.97.3:37593)
0619 04:34:04.717673 (+ 9800us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:04.717736 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:04.721165 (+ 3429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:04.725595 (+ 4430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:04.725606 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:04.726403 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:04.727071 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:04.727093 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.728688 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.728695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:04.733484 (+ 4789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:04.733498 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.734665 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.734672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:04.734850 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:04.735407 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:04.735438 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:04.741538 (+ 6100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:04.745290 (+ 3752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:04.745294 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:04.745297 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:04.746218 (+ 921us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:04.746594 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:04.746598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:04.746601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:04.746732 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:04.747365 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:04.747370 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:04.747497 (+ 127us) client_negotiation.cc:769] Sending connection context
0619 04:34:04.747797 (+ 300us) client_negotiation.cc:240] Negotiation successful
0619 04:34:04.748353 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9475,"mutex_wait_us":2685,"thread_start_us":15307,"threads_started":1}
I20260619 04:34:04.750140 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.708065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.97.3:37623)
0619 04:34:04.722556 (+ 14491us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:04.722579 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:04.722720 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:04.732471 (+ 9751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:04.732477 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:04.733087 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:04.733401 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:04.733413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.735988 (+ 2575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.736003 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:04.737738 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:04.737754 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.740895 (+ 3141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.740902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:04.741089 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:04.741748 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:04.741769 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:04.744183 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:04.747731 (+ 3548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:04.747736 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:04.747739 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:04.748157 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:04.748538 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:04.748543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:04.748545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:04.748692 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:04.749224 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:04.749228 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:04.749353 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 04:34:04.749511 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 04:34:04.749757 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14338,"spinlock_wait_cycles":37632,"thread_start_us":9152,"threads_started":1}
I20260619 04:34:04.750789 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.708371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39847 (local address 127.0.97.3:35427)
0619 04:34:04.723696 (+ 15325us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:04.723714 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:04.723817 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:04.726121 (+ 2304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:04.726125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:04.726747 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:04.727086 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:04.727101 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.729106 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.729112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:04.730882 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:04.730898 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.733766 (+ 2868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.733772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:04.734078 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:04.736130 (+ 2052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:04.736157 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:04.739235 (+ 3078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:04.745240 (+ 6005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:04.745250 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:04.745271 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:04.745788 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:04.746207 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:04.746211 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:04.746214 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:04.746420 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:04.747053 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:04.747062 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:04.747507 (+ 445us) client_negotiation.cc:769] Sending connection context
0619 04:34:04.748301 (+ 794us) client_negotiation.cc:240] Negotiation successful
0619 04:34:04.748867 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15217,"mutex_wait_us":1,"spinlock_wait_cycles":7552,"thread_start_us":14438,"threads_started":1}
I20260619 04:34:04.751129 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.707795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37593 (local address 127.0.97.60:39307)
0619 04:34:04.725064 (+ 17269us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:04.725070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:04.725092 (+ 22us) server_negotiation.cc:407] Connection header received
0619 04:34:04.725157 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:04.725163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:04.725247 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:04.725363 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:04.727245 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.728446 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.733619 (+ 5173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.734280 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.734403 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:04.741772 (+ 7369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:04.741778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:04.741783 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:04.741820 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:04.745141 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:04.746343 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:04.746348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:04.746351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:04.746479 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:04.746847 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:04.746852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:04.746855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:04.747193 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:04.747305 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:04.750415 (+ 3110us) server_negotiation.cc:299] Negotiation successful
0619 04:34:04.750643 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17150,"thread_start_us":17076,"threads_started":1}
I20260619 04:34:04.751747 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.707493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35427 (local address 127.0.97.61:39847)
0619 04:34:04.725770 (+ 18277us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:04.725776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:04.725795 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:34:04.725850 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:04.725854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:04.725937 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:04.727237 (+ 1300us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:04.727381 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.728896 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.731100 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.731724 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.734514 (+ 2790us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:04.739492 (+ 4978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:04.739499 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:04.739504 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:04.739554 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:04.745085 (+ 5531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:04.745906 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:04.745911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:04.745915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:04.746079 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:04.746577 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:04.746581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:04.746584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:04.746918 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:04.751031 (+ 4113us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:04.751172 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 04:34:04.751385 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18159,"thread_start_us":18108,"threads_started":1}
I20260619 04:34:04.753048 7798 heartbeater.cc:347] Connected to a master server at 127.0.97.61:39847
I20260619 04:34:04.753376 7798 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:04.754236 7798 heartbeater.cc:511] Master 127.0.97.61:39847 requested a full tablet report, sending...
I20260619 04:34:04.756143 7142 ts_manager.cc:194] Registered new tserver with Master: 7ec7bb13dce74266a393d0693e2b6ff5 (127.0.97.3:42411)
I20260619 04:34:04.757392 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.707234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37623 (local address 127.0.97.62:42529)
0619 04:34:04.732146 (+ 24912us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:04.732152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:04.732172 (+ 20us) server_negotiation.cc:407] Connection header received
0619 04:34:04.732230 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:04.732235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:04.732307 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:04.732413 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:04.734726 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.735865 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.738076 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.740732 (+ 2656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.740858 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:04.744405 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:04.744412 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:04.744417 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:04.744483 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:04.747625 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:04.748267 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:04.748272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:04.748276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:04.748413 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:04.748796 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:04.748800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:04.748804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:04.749119 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:04.749245 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:04.749613 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 04:34:04.749824 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24774,"thread_start_us":29534,"threads_started":1}
I20260619 04:34:04.757710 7796 heartbeater.cc:347] Connected to a master server at 127.0.97.60:39307
I20260619 04:34:04.758478 7797 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42529
I20260619 04:34:04.758579 7797 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:04.759371 7797 heartbeater.cc:511] Master 127.0.97.62:42529 requested a full tablet report, sending...
I20260619 04:34:04.759830 7796 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:04.760268 7796 heartbeater.cc:511] Master 127.0.97.60:39307 requested a full tablet report, sending...
I20260619 04:34:04.760296 6545 ts_manager.cc:194] Re-registered known tserver with Master: 7ec7bb13dce74266a393d0693e2b6ff5 (127.0.97.3:42411)
I20260619 04:34:04.761345 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37623
I20260619 04:34:04.761711 7332 ts_manager.cc:194] Registered new tserver with Master: 7ec7bb13dce74266a393d0693e2b6ff5 (127.0.97.3:42411)
I20260619 04:34:04.768795 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.730185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.0.97.61:39847)
0619 04:34:04.752422 (+ 22237us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:04.752428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:04.752445 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:34:04.752521 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:04.752526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:04.752591 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:04.752682 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:04.754728 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.755798 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.757620 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.758181 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.758282 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:04.762210 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:04.762217 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:04.762222 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:04.762263 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:04.765667 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:04.766510 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:04.766516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:04.766520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:04.766669 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:04.767074 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:04.767080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:04.767083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:04.767395 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:04.767658 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:04.768182 (+ 524us) server_negotiation.cc:299] Negotiation successful
0619 04:34:04.768400 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21687,"threads_started":1}
I20260619 04:34:04.771797 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.741560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41582 (local address 127.0.97.60:39307)
0619 04:34:04.751352 (+ 9792us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:04.751357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:04.751372 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:34:04.751416 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:04.751420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:04.751528 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:04.751621 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:04.754251 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.755292 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.757329 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.757941 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.758065 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:04.763464 (+ 5399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:04.763470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:04.763475 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:04.763511 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:04.768904 (+ 5393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:04.769666 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:04.769671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:04.769674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:04.769819 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:04.770210 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:04.770215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:04.770219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:04.770565 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:04.770794 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:04.771244 (+ 450us) server_negotiation.cc:299] Negotiation successful
0619 04:34:04.771460 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9694,"thread_start_us":26316,"threads_started":1}
I20260619 04:34:04.775990 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.727644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58020 (local address 127.0.97.62:42529)
0619 04:34:04.758010 (+ 30366us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:04.758015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:04.758030 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:34:04.758068 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:04.758073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:04.758141 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:04.758237 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:04.764288 (+ 6051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.765297 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.766722 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.767283 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.767399 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:04.770853 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:04.770859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:04.770863 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:04.770896 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:04.773863 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:04.774453 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:04.774457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:04.774459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:04.774582 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:04.774868 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:04.774872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:04.774874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:04.775119 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:04.775288 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:04.775520 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 04:34:04.775675 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30244,"thread_start_us":42934,"threads_started":1}
I20260619 04:34:04.779371 6545 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:34:04.786161 6545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:34:04.820611 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.805572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57040 (local address 127.0.97.2:37011)
0619 04:34:04.806324 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:04.806537 (+ 213us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:04.806571 (+ 34us) server_negotiation.cc:407] Connection header received
0619 04:34:04.807384 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:04.807394 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:04.807727 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:04.807940 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:04.809299 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.812195 (+ 2896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.816146 (+ 3951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.817732 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.817863 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:04.818095 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:04.820130 (+ 2035us) server_negotiation.cc:299] Negotiation successful
0619 04:34:04.820319 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":728,"threads_started":1}
I20260619 04:34:04.820930 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.804803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37011 (local address 127.0.0.1:57040)
0619 04:34:04.805439 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:04.805461 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:04.805593 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:04.808160 (+ 2567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:04.808166 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:04.808194 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:04.809102 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:04.809115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.812396 (+ 3281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.812401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:04.815948 (+ 3547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:04.815965 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.818225 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.818231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:04.818415 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:04.818453 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 04:34:04.818554 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 04:34:04.818706 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":332,"threads_started":1}
I20260619 04:34:04.824399 7590 tablet_service.cc:1467] Processing CreateTablet for tablet fb50dd9fefc846cd9a7b50897094976c (DEFAULT_TABLE table=test-table [id=65d97a40202748b68c4d66045f05c1ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:34:04.825925 7590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb50dd9fefc846cd9a7b50897094976c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:04.833689 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.805166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38759 (local address 127.0.0.1:52768)
0619 04:34:04.805734 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:04.805753 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:04.805855 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:04.828012 (+ 22157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:04.828016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:04.828042 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:04.828342 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:04.828353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.829999 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.830004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:04.831378 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:04.831389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.832875 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.832882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:04.833068 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:04.833105 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 04:34:04.833191 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 04:34:04.833320 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":297,"threads_started":1}
I20260619 04:34:04.833786 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.805766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42411 (local address 127.0.0.1:51978)
0619 04:34:04.820558 (+ 14792us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:04.820577 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:04.820689 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:04.826922 (+ 6233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:04.826928 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:04.826959 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:04.827313 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:04.827323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.828894 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.828899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:04.830288 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:04.830303 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.833113 (+ 2810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.833119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:04.833265 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:04.833294 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 04:34:04.833370 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 04:34:04.833499 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14655,"thread_start_us":14998,"threads_started":1}
I20260619 04:34:04.834261 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.806188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51978 (local address 127.0.97.3:42411)
0619 04:34:04.826295 (+ 20107us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:04.826303 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:04.826336 (+ 33us) server_negotiation.cc:407] Connection header received
0619 04:34:04.826430 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:04.826437 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:04.826695 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:04.826898 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:04.827491 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.828776 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.830452 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.831881 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.832000 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:04.832208 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:04.833443 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0619 04:34:04.833615 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19941,"thread_start_us":19746,"threads_started":1}
I20260619 04:34:04.834431 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.806038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52768 (local address 127.0.97.1:38759)
0619 04:34:04.827475 (+ 21437us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:04.827484 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:04.827520 (+ 36us) server_negotiation.cc:407] Connection header received
0619 04:34:04.827604 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:04.827610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:04.827820 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:04.828001 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:04.828528 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.829845 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.831533 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:04.832682 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:04.832799 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:04.832987 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:04.834079 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0619 04:34:04.834210 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21271,"thread_start_us":21153,"threads_started":1}
I20260619 04:34:04.840179 7445 tablet_service.cc:1467] Processing CreateTablet for tablet fb50dd9fefc846cd9a7b50897094976c (DEFAULT_TABLE table=test-table [id=65d97a40202748b68c4d66045f05c1ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:34:04.840042 7731 tablet_service.cc:1467] Processing CreateTablet for tablet fb50dd9fefc846cd9a7b50897094976c (DEFAULT_TABLE table=test-table [id=65d97a40202748b68c4d66045f05c1ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:34:04.840705 7831 tablet_bootstrap.cc:492] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e: Bootstrap starting.
I20260619 04:34:04.841575 7731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb50dd9fefc846cd9a7b50897094976c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:04.842049 7445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb50dd9fefc846cd9a7b50897094976c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:04.844218 7831 tablet_bootstrap.cc:654] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:04.845223 7831 log.cc:824] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e: Log is configured to *not* fsync() on all Append() calls
I20260619 04:34:04.847532 7831 tablet_bootstrap.cc:492] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e: No bootstrap required, opened a new log
I20260619 04:34:04.847782 7831 ts_tablet_manager.cc:1397] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260619 04:34:04.854719 7831 raft_consensus.cc:348] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37011 } } peers { permanent_uuid: "6af077b208f048a2bc017df0886b3e70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38759 } } peers { permanent_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42411 } }
I20260619 04:34:04.855057 7831 raft_consensus.cc:374] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:04.855149 7831 raft_consensus.cc:729] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f5a5fd7474244c6bfb857386b7e9e0e, State: Initialized, Role: FOLLOWER
I20260619 04:34:04.855712 7831 consensus_queue.cc:260] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37011 } } peers { permanent_uuid: "6af077b208f048a2bc017df0886b3e70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38759 } } peers { permanent_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42411 } }
I20260619 04:34:04.857472 7656 heartbeater.cc:503] Master 127.0.97.62:42529 was elected leader, sending a full tablet report...
I20260619 04:34:04.859400 7831 ts_tablet_manager.cc:1428] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260619 04:34:04.863129 7834 tablet_bootstrap.cc:492] T fb50dd9fefc846cd9a7b50897094976c P 6af077b208f048a2bc017df0886b3e70: Bootstrap starting.
I20260619 04:34:04.864979 7835 tablet_bootstrap.cc:492] T fb50dd9fefc846cd9a7b50897094976c P 7ec7bb13dce74266a393d0693e2b6ff5: Bootstrap starting.
I20260619 04:34:04.867185 7834 tablet_bootstrap.cc:654] T fb50dd9fefc846cd9a7b50897094976c P 6af077b208f048a2bc017df0886b3e70: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:04.868249 7834 log.cc:824] T fb50dd9fefc846cd9a7b50897094976c P 6af077b208f048a2bc017df0886b3e70: Log is configured to *not* fsync() on all Append() calls
I20260619 04:34:04.868355 7835 tablet_bootstrap.cc:654] T fb50dd9fefc846cd9a7b50897094976c P 7ec7bb13dce74266a393d0693e2b6ff5: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:04.869704 7835 log.cc:824] T fb50dd9fefc846cd9a7b50897094976c P 7ec7bb13dce74266a393d0693e2b6ff5: Log is configured to *not* fsync() on all Append() calls
I20260619 04:34:04.871508 7834 tablet_bootstrap.cc:492] T fb50dd9fefc846cd9a7b50897094976c P 6af077b208f048a2bc017df0886b3e70: No bootstrap required, opened a new log
I20260619 04:34:04.871744 7834 ts_tablet_manager.cc:1397] T fb50dd9fefc846cd9a7b50897094976c P 6af077b208f048a2bc017df0886b3e70: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 04:34:04.872161 7835 tablet_bootstrap.cc:492] T fb50dd9fefc846cd9a7b50897094976c P 7ec7bb13dce74266a393d0693e2b6ff5: No bootstrap required, opened a new log
I20260619 04:34:04.872370 7835 ts_tablet_manager.cc:1397] T fb50dd9fefc846cd9a7b50897094976c P 7ec7bb13dce74266a393d0693e2b6ff5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
W20260619 04:34:04.876343 7658 tablet.cc:2369] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:34:04.878474 7834 raft_consensus.cc:348] T fb50dd9fefc846cd9a7b50897094976c P 6af077b208f048a2bc017df0886b3e70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37011 } } peers { permanent_uuid: "6af077b208f048a2bc017df0886b3e70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38759 } } peers { permanent_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42411 } }
I20260619 04:34:04.878831 7834 raft_consensus.cc:374] T fb50dd9fefc846cd9a7b50897094976c P 6af077b208f048a2bc017df0886b3e70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:04.878916 7834 raft_consensus.cc:729] T fb50dd9fefc846cd9a7b50897094976c P 6af077b208f048a2bc017df0886b3e70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af077b208f048a2bc017df0886b3e70, State: Initialized, Role: FOLLOWER
I20260619 04:34:04.879448 7834 consensus_queue.cc:260] T fb50dd9fefc846cd9a7b50897094976c P 6af077b208f048a2bc017df0886b3e70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37011 } } peers { permanent_uuid: "6af077b208f048a2bc017df0886b3e70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38759 } } peers { permanent_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42411 } }
I20260619 04:34:04.879791 7835 raft_consensus.cc:348] T fb50dd9fefc846cd9a7b50897094976c P 7ec7bb13dce74266a393d0693e2b6ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37011 } } peers { permanent_uuid: "6af077b208f048a2bc017df0886b3e70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38759 } } peers { permanent_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42411 } }
I20260619 04:34:04.880123 7835 raft_consensus.cc:374] T fb50dd9fefc846cd9a7b50897094976c P 7ec7bb13dce74266a393d0693e2b6ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:04.880211 7835 raft_consensus.cc:729] T fb50dd9fefc846cd9a7b50897094976c P 7ec7bb13dce74266a393d0693e2b6ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec7bb13dce74266a393d0693e2b6ff5, State: Initialized, Role: FOLLOWER
I20260619 04:34:04.880844 7835 consensus_queue.cc:260] T fb50dd9fefc846cd9a7b50897094976c P 7ec7bb13dce74266a393d0693e2b6ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37011 } } peers { permanent_uuid: "6af077b208f048a2bc017df0886b3e70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38759 } } peers { permanent_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42411 } }
I20260619 04:34:04.881171 7834 ts_tablet_manager.cc:1428] T fb50dd9fefc846cd9a7b50897094976c P 6af077b208f048a2bc017df0886b3e70: Time spent starting tablet: real 0.009s user 0.006s sys 0.005s
I20260619 04:34:04.882913 7835 ts_tablet_manager.cc:1428] T fb50dd9fefc846cd9a7b50897094976c P 7ec7bb13dce74266a393d0693e2b6ff5: Time spent starting tablet: real 0.010s user 0.004s sys 0.007s
I20260619 04:34:04.883504 7797 heartbeater.cc:503] Master 127.0.97.62:42529 was elected leader, sending a full tablet report...
W20260619 04:34:04.974071 7799 tablet.cc:2369] T fb50dd9fefc846cd9a7b50897094976c P 7ec7bb13dce74266a393d0693e2b6ff5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:34:04.995663 7833 raft_consensus.cc:482] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:34:04.995967 7833 raft_consensus.cc:504] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37011 } } peers { permanent_uuid: "6af077b208f048a2bc017df0886b3e70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38759 } } peers { permanent_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42411 } }
I20260619 04:34:04.997640 7833 leader_election.cc:288] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af077b208f048a2bc017df0886b3e70 (127.0.97.1:38759), 7ec7bb13dce74266a393d0693e2b6ff5 (127.0.97.3:42411)
I20260619 04:34:05.004771 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.998686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55549 (local address 127.0.97.3:42411)
0619 04:34:04.998836 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:04.998840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:04.999746 (+ 906us) server_negotiation.cc:407] Connection header received
0619 04:34:04.999926 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:04.999941 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:05.000018 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:05.000095 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:05.000625 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:05.001494 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:05.003172 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:05.003938 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:05.004053 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:05.004150 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:05.004429 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 04:34:05.004574 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 04:34:05.004773 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.997946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40523 (local address 127.0.97.1:38759)
0619 04:34:04.998152 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:04.998159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:04.998564 (+ 405us) server_negotiation.cc:407] Connection header received
0619 04:34:04.998761 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:04.998767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:04.998842 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:04.998928 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:05.000026 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:05.001024 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:05.003000 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:05.003821 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:05.003939 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:05.004024 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:05.004349 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 04:34:05.004514 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 04:34:05.004894 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.997834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38759 (local address 127.0.97.2:40523)
0619 04:34:04.998437 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:04.998460 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:04.998628 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:04.999004 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:04.999008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:04.999035 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:04.999876 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:04.999889 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:05.001154 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:05.001160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:05.002859 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:05.002870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:05.004023 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:05.004029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:05.004214 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:05.004238 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 04:34:05.004306 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 04:34:05.004411 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":51,"thread_start_us":307,"threads_started":1}
I20260619 04:34:05.005009 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:04.998477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42411 (local address 127.0.97.2:55549)
0619 04:34:04.999592 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:04.999609 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:04.999804 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:05.000154 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:05.000158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:05.000182 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:05.000487 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:05.000498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:05.001619 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:05.001623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:05.003038 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:05.003051 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:05.004144 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:05.004151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:05.004273 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:05.004301 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 04:34:05.004388 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 04:34:05.004529 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":15,"thread_start_us":366,"threads_started":1}
I20260619 04:34:05.005959 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb50dd9fefc846cd9a7b50897094976c" candidate_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af077b208f048a2bc017df0886b3e70" is_pre_election: true
I20260619 04:34:05.006379 7465 raft_consensus.cc:2454] T fb50dd9fefc846cd9a7b50897094976c P 6af077b208f048a2bc017df0886b3e70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f5a5fd7474244c6bfb857386b7e9e0e in term 0.
I20260619 04:34:05.007010 7544 leader_election.cc:302] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af077b208f048a2bc017df0886b3e70, 6f5a5fd7474244c6bfb857386b7e9e0e; no voters:
I20260619 04:34:05.007136 7751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb50dd9fefc846cd9a7b50897094976c" candidate_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" is_pre_election: true
I20260619 04:34:05.007483 7833 raft_consensus.cc:2781] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:34:05.007630 7833 raft_consensus.cc:482] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:34:05.007620 7751 raft_consensus.cc:2454] T fb50dd9fefc846cd9a7b50897094976c P 7ec7bb13dce74266a393d0693e2b6ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f5a5fd7474244c6bfb857386b7e9e0e in term 0.
I20260619 04:34:05.007730 7833 raft_consensus.cc:3037] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:05.010434 7833 raft_consensus.cc:504] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37011 } } peers { permanent_uuid: "6af077b208f048a2bc017df0886b3e70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38759 } } peers { permanent_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42411 } }
I20260619 04:34:05.010979 7833 leader_election.cc:288] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e [CANDIDATE]: Term 1 election: Requested vote from peers 6af077b208f048a2bc017df0886b3e70 (127.0.97.1:38759), 7ec7bb13dce74266a393d0693e2b6ff5 (127.0.97.3:42411)
I20260619 04:34:05.011581 7751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb50dd9fefc846cd9a7b50897094976c" candidate_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec7bb13dce74266a393d0693e2b6ff5"
I20260619 04:34:05.011585 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb50dd9fefc846cd9a7b50897094976c" candidate_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af077b208f048a2bc017df0886b3e70"
I20260619 04:34:05.011782 7751 raft_consensus.cc:3037] T fb50dd9fefc846cd9a7b50897094976c P 7ec7bb13dce74266a393d0693e2b6ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:05.011803 7465 raft_consensus.cc:3037] T fb50dd9fefc846cd9a7b50897094976c P 6af077b208f048a2bc017df0886b3e70 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:05.014221 7465 raft_consensus.cc:2454] T fb50dd9fefc846cd9a7b50897094976c P 6af077b208f048a2bc017df0886b3e70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f5a5fd7474244c6bfb857386b7e9e0e in term 1.
I20260619 04:34:05.014276 7751 raft_consensus.cc:2454] T fb50dd9fefc846cd9a7b50897094976c P 7ec7bb13dce74266a393d0693e2b6ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f5a5fd7474244c6bfb857386b7e9e0e in term 1.
I20260619 04:34:05.014680 7544 leader_election.cc:302] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f5a5fd7474244c6bfb857386b7e9e0e, 7ec7bb13dce74266a393d0693e2b6ff5; no voters:
I20260619 04:34:05.014987 7833 raft_consensus.cc:2781] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:34:05.015494 7833 raft_consensus.cc:686] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e [term 1 LEADER]: Becoming Leader. State: Replica: 6f5a5fd7474244c6bfb857386b7e9e0e, State: Running, Role: LEADER
I20260619 04:34:05.016108 7833 consensus_queue.cc:237] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37011 } } peers { permanent_uuid: "6af077b208f048a2bc017df0886b3e70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38759 } } peers { permanent_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42411 } }
I20260619 04:34:05.021592 6544 catalog_manager.cc:5495] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f5a5fd7474244c6bfb857386b7e9e0e (127.0.97.2). New cstate: current_term: 1 leader_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6af077b208f048a2bc017df0886b3e70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42411 } health_report { overall_health: UNKNOWN } } }
I20260619 04:34:05.051311 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:05.046442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.0.97.2:37011)
0619 04:34:05.046640 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:05.046645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:05.046659 (+ 14us) server_negotiation.cc:407] Connection header received
0619 04:34:05.046739 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:05.046742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:05.046811 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:05.046937 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:34:05.047271 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:05.047960 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:05.049354 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:05.049743 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:05.049852 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:05.050289 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:34:05.050774 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:34:05.050910 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:05.050994 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 04:34:05.051070 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:34:05.058874 7751 raft_consensus.cc:1261] T fb50dd9fefc846cd9a7b50897094976c P 7ec7bb13dce74266a393d0693e2b6ff5 [term 1 FOLLOWER]: Refusing update from remote peer 6f5a5fd7474244c6bfb857386b7e9e0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:34:05.058947 7465 raft_consensus.cc:1261] T fb50dd9fefc846cd9a7b50897094976c P 6af077b208f048a2bc017df0886b3e70 [term 1 FOLLOWER]: Refusing update from remote peer 6f5a5fd7474244c6bfb857386b7e9e0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:34:05.059733 7842 consensus_queue.cc:1035] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:34:05.060067 7833 consensus_queue.cc:1035] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af077b208f048a2bc017df0886b3e70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 04:34:05.061040 7517 tablet.cc:2369] T fb50dd9fefc846cd9a7b50897094976c P 6af077b208f048a2bc017df0886b3e70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:34:05.071657 7845 mvcc.cc:205] Tried to move back new op lower bound from 7298431570152230912 to 7298431569990742016. Current Snapshot: MvccSnapshot[applied={T|T < 7298431570152230912}]
I20260619 04:34:05.073423 7847 mvcc.cc:205] Tried to move back new op lower bound from 7298431570152230912 to 7298431569990742016. Current Snapshot: MvccSnapshot[applied={T|T < 7298431570152230912}]
I20260619 04:34:05.088425 6545 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:34:05.088753 6545 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:34:05.095235 6545 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71decb2899a44f1fb9b71a06ff359c0f: Sending DeleteTablet for 3 replicas of tablet fb50dd9fefc846cd9a7b50897094976c
I20260619 04:34:05.096498 7445 tablet_service.cc:1514] Processing DeleteTablet for tablet fb50dd9fefc846cd9a7b50897094976c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:34:05 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52768
I20260619 04:34:05.096568 7590 tablet_service.cc:1514] Processing DeleteTablet for tablet fb50dd9fefc846cd9a7b50897094976c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:34:05 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57040
I20260619 04:34:05.096549 7731 tablet_service.cc:1514] Processing DeleteTablet for tablet fb50dd9fefc846cd9a7b50897094976c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:34:05 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51978
I20260619 04:34:05.104885 7854 tablet_replica.cc:330] T fb50dd9fefc846cd9a7b50897094976c P 6af077b208f048a2bc017df0886b3e70: stopping tablet replica
I20260619 04:34:05.105882 7854 raft_consensus.cc:2229] T fb50dd9fefc846cd9a7b50897094976c P 6af077b208f048a2bc017df0886b3e70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:34:05.106441 7854 raft_consensus.cc:2258] T fb50dd9fefc846cd9a7b50897094976c P 6af077b208f048a2bc017df0886b3e70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:34:05.107491 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 6513
I20260619 04:34:05.108573 6575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:05.109725 7855 tablet_replica.cc:330] T fb50dd9fefc846cd9a7b50897094976c P 7ec7bb13dce74266a393d0693e2b6ff5: stopping tablet replica
I20260619 04:34:05.110320 7856 tablet_replica.cc:330] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e: stopping tablet replica
I20260619 04:34:05.110859 7856 raft_consensus.cc:2229] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:34:05.111755 7856 raft_consensus.cc:2258] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:34:05.112308 7855 raft_consensus.cc:2229] T fb50dd9fefc846cd9a7b50897094976c P 7ec7bb13dce74266a393d0693e2b6ff5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:34:05.112984 7855 raft_consensus.cc:2258] T fb50dd9fefc846cd9a7b50897094976c P 7ec7bb13dce74266a393d0693e2b6ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:34:05.117218 7854 ts_tablet_manager.cc:1905] T fb50dd9fefc846cd9a7b50897094976c P 6af077b208f048a2bc017df0886b3e70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:05.125161 7854 ts_tablet_manager.cc:1918] T fb50dd9fefc846cd9a7b50897094976c P 6af077b208f048a2bc017df0886b3e70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:34:05.125437 7854 log.cc:1196] T fb50dd9fefc846cd9a7b50897094976c P 6af077b208f048a2bc017df0886b3e70: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/wal/wals/fb50dd9fefc846cd9a7b50897094976c
I20260619 04:34:05.126060 7854 ts_tablet_manager.cc:1939] T fb50dd9fefc846cd9a7b50897094976c P 6af077b208f048a2bc017df0886b3e70: Deleting consensus metadata
I20260619 04:34:05.126354 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:05.100278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43428 (local address 127.0.97.61:39847)
0619 04:34:05.100482 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:05.100488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:05.105935 (+ 5447us) server_negotiation.cc:407] Connection header received
0619 04:34:05.106218 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:05.106224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:05.106300 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:05.106374 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:05.111137 (+ 4763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:05.112567 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:05.115140 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:05.115859 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:05.116016 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:05.119706 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:05.119714 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:05.119718 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:05.119760 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:05.123195 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:05.124078 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:05.124084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:05.124088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:05.124232 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:05.124659 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:05.124663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:05.124666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:05.125017 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:05.125234 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:05.125791 (+ 557us) server_negotiation.cc:299] Negotiation successful
0619 04:34:05.126021 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 04:34:05.129863 7856 ts_tablet_manager.cc:1905] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:05.133770 7855 ts_tablet_manager.cc:1905] T fb50dd9fefc846cd9a7b50897094976c P 7ec7bb13dce74266a393d0693e2b6ff5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:05.137295 7856 ts_tablet_manager.cc:1918] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:34:05.137549 7856 log.cc:1196] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/fb50dd9fefc846cd9a7b50897094976c
I20260619 04:34:05.138201 7856 ts_tablet_manager.cc:1939] T fb50dd9fefc846cd9a7b50897094976c P 6f5a5fd7474244c6bfb857386b7e9e0e: Deleting consensus metadata
I20260619 04:34:05.141952 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:05.106196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.0.97.60:39307)
0619 04:34:05.106377 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:05.106384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:05.127640 (+ 21256us) server_negotiation.cc:407] Connection header received
0619 04:34:05.127898 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:05.127904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:05.127974 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:05.128058 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:05.129455 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:05.130215 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:05.131808 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:05.132190 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:05.132296 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:05.135411 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:05.135418 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:05.135422 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:05.135542 (+ 120us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:05.138972 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:05.139754 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:05.139760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:05.139764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:05.139904 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:05.140333 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:05.140338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:05.140342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:05.140682 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:05.140882 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:05.141398 (+ 516us) server_negotiation.cc:299] Negotiation successful
0619 04:34:05.141603 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 04:34:05.153992 7855 ts_tablet_manager.cc:1918] T fb50dd9fefc846cd9a7b50897094976c P 7ec7bb13dce74266a393d0693e2b6ff5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:34:05.154155 7855 log.cc:1196] T fb50dd9fefc846cd9a7b50897094976c P 7ec7bb13dce74266a393d0693e2b6ff5: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/fb50dd9fefc846cd9a7b50897094976c
I20260619 04:34:05.154533 7855 ts_tablet_manager.cc:1939] T fb50dd9fefc846cd9a7b50897094976c P 7ec7bb13dce74266a393d0693e2b6ff5: Deleting consensus metadata
I20260619 04:34:05.260737 6530 catalog_manager.cc:4841] TS 7ec7bb13dce74266a393d0693e2b6ff5 (127.0.97.3:42411): tablet fb50dd9fefc846cd9a7b50897094976c (table test-table [id=65d97a40202748b68c4d66045f05c1ff]) successfully deleted
I20260619 04:34:05.261104 6530 catalog_manager.cc:4841] TS 6af077b208f048a2bc017df0886b3e70 (127.0.97.1:38759): tablet fb50dd9fefc846cd9a7b50897094976c (table test-table [id=65d97a40202748b68c4d66045f05c1ff]) successfully deleted
I20260619 04:34:05.261345 6529 catalog_manager.cc:4841] TS 6f5a5fd7474244c6bfb857386b7e9e0e (127.0.97.2:37011): tablet fb50dd9fefc846cd9a7b50897094976c (table test-table [id=65d97a40202748b68c4d66045f05c1ff]) successfully deleted
I20260619 04:34:05.261407 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 6513
I20260619 04:34:05.315486 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:05.293307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41604 (local address 127.0.97.60:39307)
0619 04:34:05.293460 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:05.293465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:05.296344 (+ 2879us) server_negotiation.cc:407] Connection header received
0619 04:34:05.296889 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:05.296897 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:05.296988 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:05.297065 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:05.299316 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:05.300515 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:05.303466 (+ 2951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:05.304111 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:05.304212 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:05.307805 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:05.307812 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:05.307817 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:05.307861 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:05.311586 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:05.312313 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:05.312319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:05.312323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:05.312443 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:05.312794 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:05.312799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:05.312803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:05.313130 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:05.313429 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:05.314989 (+ 1560us) server_negotiation.cc:299] Negotiation successful
0619 04:34:05.315170 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 04:34:05.317199 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:05.293257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.0.97.61:39847)
0619 04:34:05.293436 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:05.293442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:05.303044 (+ 9602us) server_negotiation.cc:407] Connection header received
0619 04:34:05.303252 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:05.303258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:05.303346 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:05.303417 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:05.305903 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:05.307148 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:05.308748 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:05.309201 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:05.309315 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:05.311505 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:05.311510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:05.311513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:05.311545 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:05.314332 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:05.315037 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:05.315042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:05.315046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:05.315186 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:05.315549 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:05.315553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:05.315556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:05.315853 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:05.316062 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:05.316594 (+ 532us) server_negotiation.cc:299] Negotiation successful
0619 04:34:05.316803 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 04:34:06.083302 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.081898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.97.2:34911)
0619 04:34:06.082906 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.083083 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":799,"threads_started":1}
W20260619 04:34:06.083863 7656 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42529 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
I20260619 04:34:06.084031 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.081894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.97.1:35875)
0619 04:34:06.083748 (+ 1854us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.083871 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1731,"thread_start_us":1630,"threads_started":1}
I20260619 04:34:06.084538 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.082974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.97.3:33323)
0619 04:34:06.084043 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.084166 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":955,"thread_start_us":1213,"threads_started":1}
W20260619 04:34:06.084699 7511 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42529 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
W20260619 04:34:06.084892 7797 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42529 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
I20260619 04:34:06.596632 7869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71decb2899a44f1fb9b71a06ff359c0f)
I20260619 04:34:06.596910 7869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: false } }
I20260619 04:34:06.597224 7870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71decb2899a44f1fb9b71a06ff359c0f)
I20260619 04:34:06.597399 7870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: false } }
I20260619 04:34:06.598129 7870 leader_election.cc:288] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71decb2899a44f1fb9b71a06ff359c0f (127.0.97.62:42529), 7a74506f385f403d9104e5dcdb2c851a (127.0.97.61:39847)
I20260619 04:34:06.599255 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.598236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.0.1:58430)
0619 04:34:06.598709 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.598893 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":225,"threads_started":1}
W20260619 04:34:06.599658 7316 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
I20260619 04:34:06.610875 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.610069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.0.1:58450)
0619 04:34:06.610528 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.610654 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341}
I20260619 04:34:06.613157 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.598333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.0.1:58444)
0619 04:34:06.611900 (+ 13567us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.612110 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13429,"threads_started":1}
W20260619 04:34:06.613461 7316 leader_election.cc:334] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71decb2899a44f1fb9b71a06ff359c0f (127.0.97.62:42529): Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
W20260619 04:34:06.614447 7126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
I20260619 04:34:06.616739 7869 leader_election.cc:288] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71decb2899a44f1fb9b71a06ff359c0f (127.0.97.62:42529), 974acf9a4cd54e84b8fe000bf11137a8 (127.0.97.60:39307)
I20260619 04:34:06.617231 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.598353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39847 (local address 127.0.0.1:43434)
0619 04:34:06.605070 (+ 6717us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.605093 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:06.605274 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:06.610931 (+ 5657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:06.610936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:06.610979 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:06.611287 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:06.611303 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.613186 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.613191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:06.615157 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:06.615168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.616485 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.616491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:06.616661 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:06.616694 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 04:34:06.616789 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 04:34:06.616922 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6597,"thread_start_us":12813,"threads_started":1}
I20260619 04:34:06.617905 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.598789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43434 (local address 127.0.97.61:39847)
0619 04:34:06.610584 (+ 11795us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:06.610591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:06.610611 (+ 20us) server_negotiation.cc:407] Connection header received
0619 04:34:06.610680 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:06.610685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:06.610766 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:06.611474 (+ 708us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:06.611734 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.612999 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.615333 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.616290 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.616397 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:06.616576 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:06.617108 (+ 532us) server_negotiation.cc:299] Negotiation successful
0619 04:34:06.617315 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11682,"thread_start_us":15385,"threads_started":1}
I20260619 04:34:06.618301 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.616855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.0.1:58454)
0619 04:34:06.617872 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.618027 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":852}
W20260619 04:34:06.618499 7126 leader_election.cc:334] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71decb2899a44f1fb9b71a06ff359c0f (127.0.97.62:42529): Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
I20260619 04:34:06.618546 7152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "974acf9a4cd54e84b8fe000bf11137a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7a74506f385f403d9104e5dcdb2c851a" is_pre_election: true
I20260619 04:34:06.618785 7152 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974acf9a4cd54e84b8fe000bf11137a8 in term 1.
I20260619 04:34:06.619196 7319 leader_election.cc:302] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a74506f385f403d9104e5dcdb2c851a, 974acf9a4cd54e84b8fe000bf11137a8; no voters: 71decb2899a44f1fb9b71a06ff359c0f
I20260619 04:34:06.619474 7870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 04:34:06.619593 7870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71decb2899a44f1fb9b71a06ff359c0f)
I20260619 04:34:06.619683 7870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 1 FOLLOWER]: Advancing to term 2
I20260619 04:34:06.622287 7870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: false } }
I20260619 04:34:06.622645 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.610886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39307 (local address 127.0.0.1:41618)
0619 04:34:06.614280 (+ 3394us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.614296 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:06.614395 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:06.615143 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:06.615147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:06.615176 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:06.617673 (+ 2497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:06.617690 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.619010 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.619015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:06.620847 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:06.620861 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.621983 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.621990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:06.622146 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:06.622178 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 04:34:06.622268 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 04:34:06.622395 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3303,"thread_start_us":6362,"threads_started":1}
I20260619 04:34:06.622695 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.611449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41618 (local address 127.0.97.60:39307)
0619 04:34:06.614665 (+ 3216us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:06.614670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:06.614691 (+ 21us) server_negotiation.cc:407] Connection header received
0619 04:34:06.614740 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:06.614744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:06.614810 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:06.614895 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:06.617890 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.618903 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.620994 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.621824 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.621915 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:06.621991 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:06.622300 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 04:34:06.622385 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3090,"thread_start_us":3029,"threads_started":1}
I20260619 04:34:06.622958 7870 leader_election.cc:288] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [CANDIDATE]: Term 2 election: Requested vote from peers 71decb2899a44f1fb9b71a06ff359c0f (127.0.97.62:42529), 7a74506f385f403d9104e5dcdb2c851a (127.0.97.61:39847)
I20260619 04:34:06.623404 7342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a74506f385f403d9104e5dcdb2c851a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "974acf9a4cd54e84b8fe000bf11137a8" is_pre_election: true
I20260619 04:34:06.623557 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.623088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.0.1:58458)
0619 04:34:06.623303 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.623380 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260619 04:34:06.623770 7342 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a74506f385f403d9104e5dcdb2c851a in current term 2: Already voted for candidate 974acf9a4cd54e84b8fe000bf11137a8 in this term.
I20260619 04:34:06.624150 7152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "974acf9a4cd54e84b8fe000bf11137a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7a74506f385f403d9104e5dcdb2c851a"
I20260619 04:34:06.624392 7152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 1 FOLLOWER]: Advancing to term 2
I20260619 04:34:06.624348 7127 leader_election.cc:302] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a74506f385f403d9104e5dcdb2c851a; no voters: 71decb2899a44f1fb9b71a06ff359c0f, 974acf9a4cd54e84b8fe000bf11137a8
I20260619 04:34:06.624476 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.624071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.0.1:58460)
0619 04:34:06.624238 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.624324 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260619 04:34:06.624647 7316 leader_election.cc:334] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71decb2899a44f1fb9b71a06ff359c0f (127.0.97.62:42529): Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
I20260619 04:34:06.626384 7152 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974acf9a4cd54e84b8fe000bf11137a8 in term 2.
I20260619 04:34:06.626533 7869 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 04:34:06.626772 7319 leader_election.cc:302] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a74506f385f403d9104e5dcdb2c851a, 974acf9a4cd54e84b8fe000bf11137a8; no voters: 71decb2899a44f1fb9b71a06ff359c0f
I20260619 04:34:06.627043 7870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 04:34:06.627422 7870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [term 2 LEADER]: Becoming Leader. State: Replica: 974acf9a4cd54e84b8fe000bf11137a8, State: Running, Role: LEADER
I20260619 04:34:06.627768 7870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: false } }
I20260619 04:34:06.629570 7879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 974acf9a4cd54e84b8fe000bf11137a8. Latest consensus state: current_term: 2 leader_uuid: "974acf9a4cd54e84b8fe000bf11137a8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: false } } }
I20260619 04:34:06.629909 7879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [sys.catalog]: This master's current role is: LEADER
I20260619 04:34:06.630477 7881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:34:06.633809 7881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:34:06.634546 7881 catalog_manager.cc:1259] Loaded cluster ID: 7704a7aa6f7e4cddaf287d60a6d4102c
I20260619 04:34:06.634610 7881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:34:06.635632 7881 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:34:06.636340 7881 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8: Loaded TSK: 0
I20260619 04:34:06.637876 7881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:34:06.695950 7331 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:34:06.702608 7331 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:34:06.706594 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.706161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.0.1:58492)
0619 04:34:06.706327 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.706416 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 04:34:06.706625 7152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [term 2 FOLLOWER]: Refusing update from remote peer 974acf9a4cd54e84b8fe000bf11137a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 04:34:06.707295 7879 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 04:34:06.707456 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.707135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.0.1:58502)
0619 04:34:06.707272 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.707345 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 04:34:06.707618 7316 consensus_peers.cc:487] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 -> Peer 71decb2899a44f1fb9b71a06ff359c0f (127.0.97.62:42529): Couldn't send request to peer 71decb2899a44f1fb9b71a06ff359c0f. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 04:34:06.709753 7869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 974acf9a4cd54e84b8fe000bf11137a8. Latest consensus state: current_term: 2 leader_uuid: "974acf9a4cd54e84b8fe000bf11137a8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: false } } }
I20260619 04:34:06.709986 7869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:34:06.711736 7879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "974acf9a4cd54e84b8fe000bf11137a8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: false } } }
I20260619 04:34:06.711817 7869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "974acf9a4cd54e84b8fe000bf11137a8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71decb2899a44f1fb9b71a06ff359c0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42529 } } peers { permanent_uuid: "7a74506f385f403d9104e5dcdb2c851a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39847 } attrs { promote: false } } peers { permanent_uuid: "974acf9a4cd54e84b8fe000bf11137a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39307 } attrs { promote: false } } }
I20260619 04:34:06.711992 7879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 [sys.catalog]: This master's current role is: LEADER
I20260619 04:34:06.712066 7869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a74506f385f403d9104e5dcdb2c851a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:34:06.727348 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.719703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38759 (local address 127.0.0.1:52772)
0619 04:34:06.719848 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.719865 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:06.719971 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:06.720691 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:06.720696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:06.720719 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:06.721109 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:06.721119 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.722569 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.722576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:06.724825 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:06.724840 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.726506 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.726513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:06.726704 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:06.726744 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 04:34:06.726811 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 04:34:06.726961 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 04:34:06.727984 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.719359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42411 (local address 127.0.0.1:51980)
0619 04:34:06.719756 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.719774 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:06.719898 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:06.720256 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:06.720260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:06.720285 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:06.720481 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:06.720493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.722369 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.722375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:06.724501 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:06.724516 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.727240 (+ 2724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.727247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:06.727444 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:06.727480 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 04:34:06.727541 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 04:34:06.727678 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":4}
I20260619 04:34:06.728844 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.719990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52772 (local address 127.0.97.1:38759)
0619 04:34:06.720374 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:06.720379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:06.720395 (+ 16us) server_negotiation.cc:407] Connection header received
0619 04:34:06.720479 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:06.720483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:06.720551 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:06.721249 (+ 698us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:06.721397 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.722419 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.725028 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.726266 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.728213 (+ 1947us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:06.728332 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:06.728382 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 04:34:06.728541 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":198,"threads_started":1}
I20260619 04:34:06.729707 7445 tablet_service.cc:1467] Processing CreateTablet for tablet f738929269a54873afa9d51b5ed51ebb (DEFAULT_TABLE table=test-table [id=2048dafd5d144fd4924cc8493f665018]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:34:06.730111 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.719524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51980 (local address 127.0.97.3:42411)
0619 04:34:06.719927 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:06.719933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:06.719952 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:34:06.720016 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:06.720021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:06.720119 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:06.720617 (+ 498us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:06.720788 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.722211 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.725733 (+ 3522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.727038 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.728389 (+ 1351us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:06.728522 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:06.728574 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 04:34:06.728720 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":448,"threads_started":1}
I20260619 04:34:06.730201 7445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f738929269a54873afa9d51b5ed51ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:06.733145 7731 tablet_service.cc:1467] Processing CreateTablet for tablet f738929269a54873afa9d51b5ed51ebb (DEFAULT_TABLE table=test-table [id=2048dafd5d144fd4924cc8493f665018]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:34:06.733603 7731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f738929269a54873afa9d51b5ed51ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:06.736896 7892 tablet_bootstrap.cc:492] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70: Bootstrap starting.
I20260619 04:34:06.737547 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.720417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37011 (local address 127.0.0.1:57050)
0619 04:34:06.727569 (+ 7152us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.727589 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:06.727680 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:06.733138 (+ 5458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:06.733142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:06.733167 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:06.733298 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:06.733308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.734535 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.734539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:06.735676 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:06.735687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.736803 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.736809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:06.736998 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:06.737034 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 04:34:06.737114 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 04:34:06.737243 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7061,"thread_start_us":10664,"threads_started":1}
I20260619 04:34:06.738308 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.720584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.0.97.2:37011)
0619 04:34:06.732832 (+ 12248us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:06.732838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:06.732857 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:34:06.732922 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:06.732927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:06.732999 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:06.733097 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:06.733445 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.734390 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.735832 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.736644 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.737772 (+ 1128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:06.737868 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:06.737918 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 04:34:06.738038 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12135,"thread_start_us":12047,"threads_started":1}
I20260619 04:34:06.738526 7892 tablet_bootstrap.cc:654] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:06.738994 7590 tablet_service.cc:1467] Processing CreateTablet for tablet f738929269a54873afa9d51b5ed51ebb (DEFAULT_TABLE table=test-table [id=2048dafd5d144fd4924cc8493f665018]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:34:06.739202 7893 tablet_bootstrap.cc:492] T f738929269a54873afa9d51b5ed51ebb P 7ec7bb13dce74266a393d0693e2b6ff5: Bootstrap starting.
I20260619 04:34:06.739409 7590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f738929269a54873afa9d51b5ed51ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:06.740725 7893 tablet_bootstrap.cc:654] T f738929269a54873afa9d51b5ed51ebb P 7ec7bb13dce74266a393d0693e2b6ff5: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:06.741243 7892 tablet_bootstrap.cc:492] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70: No bootstrap required, opened a new log
I20260619 04:34:06.741379 7892 ts_tablet_manager.cc:1397] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 04:34:06.742309 7893 tablet_bootstrap.cc:492] T f738929269a54873afa9d51b5ed51ebb P 7ec7bb13dce74266a393d0693e2b6ff5: No bootstrap required, opened a new log
I20260619 04:34:06.742159 7892 raft_consensus.cc:348] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42411 } } peers { permanent_uuid: "6af077b208f048a2bc017df0886b3e70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38759 } } peers { permanent_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37011 } }
I20260619 04:34:06.742386 7892 raft_consensus.cc:374] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:06.742432 7893 ts_tablet_manager.cc:1397] T f738929269a54873afa9d51b5ed51ebb P 7ec7bb13dce74266a393d0693e2b6ff5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 04:34:06.742503 7892 raft_consensus.cc:729] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af077b208f048a2bc017df0886b3e70, State: Initialized, Role: FOLLOWER
I20260619 04:34:06.742771 7892 consensus_queue.cc:260] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42411 } } peers { permanent_uuid: "6af077b208f048a2bc017df0886b3e70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38759 } } peers { permanent_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37011 } }
I20260619 04:34:06.743116 7893 raft_consensus.cc:348] T f738929269a54873afa9d51b5ed51ebb P 7ec7bb13dce74266a393d0693e2b6ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42411 } } peers { permanent_uuid: "6af077b208f048a2bc017df0886b3e70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38759 } } peers { permanent_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37011 } }
I20260619 04:34:06.743392 7893 raft_consensus.cc:374] T f738929269a54873afa9d51b5ed51ebb P 7ec7bb13dce74266a393d0693e2b6ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:06.743520 7893 raft_consensus.cc:729] T f738929269a54873afa9d51b5ed51ebb P 7ec7bb13dce74266a393d0693e2b6ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec7bb13dce74266a393d0693e2b6ff5, State: Initialized, Role: FOLLOWER
I20260619 04:34:06.743870 7893 consensus_queue.cc:260] T f738929269a54873afa9d51b5ed51ebb P 7ec7bb13dce74266a393d0693e2b6ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42411 } } peers { permanent_uuid: "6af077b208f048a2bc017df0886b3e70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38759 } } peers { permanent_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37011 } }
I20260619 04:34:06.750758 7893 ts_tablet_manager.cc:1428] T f738929269a54873afa9d51b5ed51ebb P 7ec7bb13dce74266a393d0693e2b6ff5: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260619 04:34:06.751329 7899 tablet_bootstrap.cc:492] T f738929269a54873afa9d51b5ed51ebb P 6f5a5fd7474244c6bfb857386b7e9e0e: Bootstrap starting.
I20260619 04:34:06.753041 7899 tablet_bootstrap.cc:654] T f738929269a54873afa9d51b5ed51ebb P 6f5a5fd7474244c6bfb857386b7e9e0e: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:06.753069 7796 heartbeater.cc:503] Master 127.0.97.60:39307 was elected leader, sending a full tablet report...
I20260619 04:34:06.753352 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.751485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.97.3:44947)
0619 04:34:06.752352 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.752505 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":1274,"threads_started":1}
I20260619 04:34:06.756286 7892 ts_tablet_manager.cc:1428] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20260619 04:34:06.756616 7510 heartbeater.cc:503] Master 127.0.97.60:39307 was elected leader, sending a full tablet report...
I20260619 04:34:06.756922 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.754092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.97.3:58391)
0619 04:34:06.756625 (+ 2533us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.756773 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2411}
I20260619 04:34:06.757982 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.744265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.97.1:45563)
0619 04:34:06.757582 (+ 13317us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.757718 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13166,"threads_started":1}
I20260619 04:34:06.759262 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.758914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.97.1:51337)
0619 04:34:06.759067 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.759142 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 04:34:06.759357 7899 tablet_bootstrap.cc:492] T f738929269a54873afa9d51b5ed51ebb P 6f5a5fd7474244c6bfb857386b7e9e0e: No bootstrap required, opened a new log
I20260619 04:34:06.759490 7899 ts_tablet_manager.cc:1397] T f738929269a54873afa9d51b5ed51ebb P 6f5a5fd7474244c6bfb857386b7e9e0e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 04:34:06.760071 7899 raft_consensus.cc:348] T f738929269a54873afa9d51b5ed51ebb P 6f5a5fd7474244c6bfb857386b7e9e0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42411 } } peers { permanent_uuid: "6af077b208f048a2bc017df0886b3e70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38759 } } peers { permanent_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37011 } }
I20260619 04:34:06.760234 7899 raft_consensus.cc:374] T f738929269a54873afa9d51b5ed51ebb P 6f5a5fd7474244c6bfb857386b7e9e0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:06.760316 7899 raft_consensus.cc:729] T f738929269a54873afa9d51b5ed51ebb P 6f5a5fd7474244c6bfb857386b7e9e0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f5a5fd7474244c6bfb857386b7e9e0e, State: Initialized, Role: FOLLOWER
I20260619 04:34:06.760550 7899 consensus_queue.cc:260] T f738929269a54873afa9d51b5ed51ebb P 6f5a5fd7474244c6bfb857386b7e9e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42411 } } peers { permanent_uuid: "6af077b208f048a2bc017df0886b3e70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38759 } } peers { permanent_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37011 } }
I20260619 04:34:06.761199 7899 ts_tablet_manager.cc:1428] T f738929269a54873afa9d51b5ed51ebb P 6f5a5fd7474244c6bfb857386b7e9e0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 04:34:06.762279 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.761562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.97.2:45077)
0619 04:34:06.762014 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.762113 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":251,"threads_started":1}
I20260619 04:34:06.762740 7655 heartbeater.cc:503] Master 127.0.97.60:39307 was elected leader, sending a full tablet report...
I20260619 04:34:06.763019 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.762729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.97.2:48471)
0619 04:34:06.762856 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.762911 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 04:34:06.784107 7896 raft_consensus.cc:482] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:34:06.784296 7896 raft_consensus.cc:504] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42411 } } peers { permanent_uuid: "6af077b208f048a2bc017df0886b3e70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38759 } } peers { permanent_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37011 } }
I20260619 04:34:06.785836 7896 leader_election.cc:288] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ec7bb13dce74266a393d0693e2b6ff5 (127.0.97.3:42411), 6f5a5fd7474244c6bfb857386b7e9e0e (127.0.97.2:37011)
I20260619 04:34:06.791241 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.786139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38329 (local address 127.0.97.3:42411)
0619 04:34:06.786324 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:06.786332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:06.786356 (+ 24us) server_negotiation.cc:407] Connection header received
0619 04:34:06.786536 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:06.786540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:06.786598 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:06.786668 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:06.787133 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.787869 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.789627 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.790411 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.790520 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:06.790617 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:06.790867 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 04:34:06.790990 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 04:34:06.791301 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.786004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42411 (local address 127.0.97.1:38329)
0619 04:34:06.786267 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.786286 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:06.786408 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:06.786706 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:06.786711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:06.786754 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:06.786979 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:06.786987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.787982 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.787987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:06.789489 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:06.789500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.790560 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.790566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:06.790728 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:06.790751 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 04:34:06.790834 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 04:34:06.790978 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260619 04:34:06.791616 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.786098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37011 (local address 127.0.97.1:45203)
0619 04:34:06.786716 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.786732 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:06.786855 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:06.787136 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:06.787141 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:06.787161 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:06.787402 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:06.787409 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.788341 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.788346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:06.789696 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:06.789707 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.790878 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.790884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:06.791044 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:06.791092 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 04:34:06.791186 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 04:34:06.791275 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":405,"threads_started":1}
I20260619 04:34:06.791934 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.786254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45203 (local address 127.0.97.2:37011)
0619 04:34:06.786427 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:06.786433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:06.786816 (+ 383us) server_negotiation.cc:407] Connection header received
0619 04:34:06.786976 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:06.786980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:06.787036 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:06.787108 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:06.787547 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.788223 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.789836 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.790686 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.790795 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:06.790888 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:06.791595 (+ 707us) server_negotiation.cc:299] Negotiation successful
0619 04:34:06.791716 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:34:06.791855 7751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f738929269a54873afa9d51b5ed51ebb" candidate_uuid: "6af077b208f048a2bc017df0886b3e70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" is_pre_election: true
I20260619 04:34:06.792101 7751 raft_consensus.cc:2454] T f738929269a54873afa9d51b5ed51ebb P 7ec7bb13dce74266a393d0693e2b6ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af077b208f048a2bc017df0886b3e70 in term 0.
I20260619 04:34:06.792600 7399 leader_election.cc:302] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af077b208f048a2bc017df0886b3e70, 7ec7bb13dce74266a393d0693e2b6ff5; no voters:
I20260619 04:34:06.792692 7610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f738929269a54873afa9d51b5ed51ebb" candidate_uuid: "6af077b208f048a2bc017df0886b3e70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" is_pre_election: true
I20260619 04:34:06.792956 7896 raft_consensus.cc:2781] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:34:06.793056 7896 raft_consensus.cc:482] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:34:06.793058 7610 raft_consensus.cc:2454] T f738929269a54873afa9d51b5ed51ebb P 6f5a5fd7474244c6bfb857386b7e9e0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af077b208f048a2bc017df0886b3e70 in term 0.
I20260619 04:34:06.793143 7896 raft_consensus.cc:3037] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:06.795482 7896 raft_consensus.cc:504] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42411 } } peers { permanent_uuid: "6af077b208f048a2bc017df0886b3e70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38759 } } peers { permanent_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37011 } }
I20260619 04:34:06.796008 7896 leader_election.cc:288] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70 [CANDIDATE]: Term 1 election: Requested vote from peers 7ec7bb13dce74266a393d0693e2b6ff5 (127.0.97.3:42411), 6f5a5fd7474244c6bfb857386b7e9e0e (127.0.97.2:37011)
I20260619 04:34:06.796414 7610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f738929269a54873afa9d51b5ed51ebb" candidate_uuid: "6af077b208f048a2bc017df0886b3e70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e"
I20260619 04:34:06.796417 7751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f738929269a54873afa9d51b5ed51ebb" candidate_uuid: "6af077b208f048a2bc017df0886b3e70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec7bb13dce74266a393d0693e2b6ff5"
I20260619 04:34:06.796634 7610 raft_consensus.cc:3037] T f738929269a54873afa9d51b5ed51ebb P 6f5a5fd7474244c6bfb857386b7e9e0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:06.796651 7751 raft_consensus.cc:3037] T f738929269a54873afa9d51b5ed51ebb P 7ec7bb13dce74266a393d0693e2b6ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:06.798524 7610 raft_consensus.cc:2454] T f738929269a54873afa9d51b5ed51ebb P 6f5a5fd7474244c6bfb857386b7e9e0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af077b208f048a2bc017df0886b3e70 in term 1.
I20260619 04:34:06.798558 7751 raft_consensus.cc:2454] T f738929269a54873afa9d51b5ed51ebb P 7ec7bb13dce74266a393d0693e2b6ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af077b208f048a2bc017df0886b3e70 in term 1.
I20260619 04:34:06.798875 7398 leader_election.cc:302] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af077b208f048a2bc017df0886b3e70, 6f5a5fd7474244c6bfb857386b7e9e0e; no voters:
I20260619 04:34:06.799161 7896 raft_consensus.cc:2781] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:34:06.799590 7896 raft_consensus.cc:686] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70 [term 1 LEADER]: Becoming Leader. State: Replica: 6af077b208f048a2bc017df0886b3e70, State: Running, Role: LEADER
I20260619 04:34:06.799963 7896 consensus_queue.cc:237] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42411 } } peers { permanent_uuid: "6af077b208f048a2bc017df0886b3e70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38759 } } peers { permanent_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37011 } }
I20260619 04:34:06.800575 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.800104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.97.1:34283)
0619 04:34:06.800285 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.800393 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 04:34:06.803239 7331 catalog_manager.cc:5495] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6af077b208f048a2bc017df0886b3e70 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "6af077b208f048a2bc017df0886b3e70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6af077b208f048a2bc017df0886b3e70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37011 } health_report { overall_health: UNKNOWN } } }
I20260619 04:34:06.824554 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.820100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52782 (local address 127.0.97.1:38759)
0619 04:34:06.820266 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:06.820271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:06.820286 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:34:06.820412 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:06.820415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:06.820540 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:06.820663 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:34:06.820943 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.821632 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.822816 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.823203 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.823284 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:06.823603 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:34:06.824108 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:34:06.824205 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:06.824264 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 04:34:06.824324 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 04:34:06.828060 7751 raft_consensus.cc:1261] T f738929269a54873afa9d51b5ed51ebb P 7ec7bb13dce74266a393d0693e2b6ff5 [term 1 FOLLOWER]: Refusing update from remote peer 6af077b208f048a2bc017df0886b3e70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:34:06.828269 7610 raft_consensus.cc:1261] T f738929269a54873afa9d51b5ed51ebb P 6f5a5fd7474244c6bfb857386b7e9e0e [term 1 FOLLOWER]: Refusing update from remote peer 6af077b208f048a2bc017df0886b3e70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:34:06.828734 7905 consensus_queue.cc:1035] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec7bb13dce74266a393d0693e2b6ff5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:34:06.829082 7896 consensus_queue.cc:1035] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f5a5fd7474244c6bfb857386b7e9e0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:34:06.831611 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.831058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.97.3:53209)
0619 04:34:06.831300 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.831411 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260619 04:34:06.831625 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.831168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.97.2:37535)
0619 04:34:06.831371 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.831473 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260619 04:34:06.838186 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.837704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.97.2:60209)
0619 04:34:06.837921 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.838012 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260619 04:34:06.838510 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.838151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.97.3:45333)
0619 04:34:06.838277 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.838353 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 04:34:06.839232 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.838620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.97.1:36831)
0619 04:34:06.839005 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:06.839086 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288}
I20260619 04:34:06.840616 7909 mvcc.cc:205] Tried to move back new op lower bound from 7298431577401905152 to 7298431577296855040. Current Snapshot: MvccSnapshot[applied={T|T < 7298431577401905152}]
I20260619 04:34:06.850839 7331 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:34:06.851069 7331 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:34:06.856513 7331 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8: Sending DeleteTablet for 3 replicas of tablet f738929269a54873afa9d51b5ed51ebb
I20260619 04:34:06.857458 7731 tablet_service.cc:1514] Processing DeleteTablet for tablet f738929269a54873afa9d51b5ed51ebb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:34:06 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:51980
I20260619 04:34:06.857615 7445 tablet_service.cc:1514] Processing DeleteTablet for tablet f738929269a54873afa9d51b5ed51ebb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:34:06 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:52772
I20260619 04:34:06.858112 7590 tablet_service.cc:1514] Processing DeleteTablet for tablet f738929269a54873afa9d51b5ed51ebb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:34:06 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:57050
I20260619 04:34:06.858278 7918 tablet_replica.cc:330] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70: stopping tablet replica
I20260619 04:34:06.858497 7918 raft_consensus.cc:2229] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:34:06.859100 7918 raft_consensus.cc:2258] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:34:06.860118 7918 ts_tablet_manager.cc:1905] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:06.861150 7919 tablet_replica.cc:330] T f738929269a54873afa9d51b5ed51ebb P 6f5a5fd7474244c6bfb857386b7e9e0e: stopping tablet replica
I20260619 04:34:06.861383 7919 raft_consensus.cc:2229] T f738929269a54873afa9d51b5ed51ebb P 6f5a5fd7474244c6bfb857386b7e9e0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:34:06.861726 7919 raft_consensus.cc:2258] T f738929269a54873afa9d51b5ed51ebb P 6f5a5fd7474244c6bfb857386b7e9e0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:34:06.862181 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 7304
I20260619 04:34:06.862303 7917 tablet_replica.cc:330] T f738929269a54873afa9d51b5ed51ebb P 7ec7bb13dce74266a393d0693e2b6ff5: stopping tablet replica
I20260619 04:34:06.862461 7917 raft_consensus.cc:2229] T f738929269a54873afa9d51b5ed51ebb P 7ec7bb13dce74266a393d0693e2b6ff5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:34:06.862738 7917 raft_consensus.cc:2258] T f738929269a54873afa9d51b5ed51ebb P 7ec7bb13dce74266a393d0693e2b6ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:34:06.864069 7917 ts_tablet_manager.cc:1905] T f738929269a54873afa9d51b5ed51ebb P 7ec7bb13dce74266a393d0693e2b6ff5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:06.867314 7918 ts_tablet_manager.cc:1918] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:34:06.867496 7918 log.cc:1196] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-0/wal/wals/f738929269a54873afa9d51b5ed51ebb
I20260619 04:34:06.867991 7918 ts_tablet_manager.cc:1939] T f738929269a54873afa9d51b5ed51ebb P 6af077b208f048a2bc017df0886b3e70: Deleting consensus metadata
I20260619 04:34:06.869136 7317 catalog_manager.cc:4841] TS 6af077b208f048a2bc017df0886b3e70 (127.0.97.1:38759): tablet f738929269a54873afa9d51b5ed51ebb (table test-table [id=2048dafd5d144fd4924cc8493f665018]) successfully deleted
I20260619 04:34:06.870533 7917 ts_tablet_manager.cc:1918] T f738929269a54873afa9d51b5ed51ebb P 7ec7bb13dce74266a393d0693e2b6ff5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:34:06.870667 7917 log.cc:1196] T f738929269a54873afa9d51b5ed51ebb P 7ec7bb13dce74266a393d0693e2b6ff5: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/f738929269a54873afa9d51b5ed51ebb
I20260619 04:34:06.871083 7917 ts_tablet_manager.cc:1939] T f738929269a54873afa9d51b5ed51ebb P 7ec7bb13dce74266a393d0693e2b6ff5: Deleting consensus metadata
I20260619 04:34:06.872102 7317 catalog_manager.cc:4841] TS 7ec7bb13dce74266a393d0693e2b6ff5 (127.0.97.3:42411): tablet f738929269a54873afa9d51b5ed51ebb (table test-table [id=2048dafd5d144fd4924cc8493f665018]) successfully deleted
I20260619 04:34:06.872913 7919 ts_tablet_manager.cc:1905] T f738929269a54873afa9d51b5ed51ebb P 6f5a5fd7474244c6bfb857386b7e9e0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:06.877646 7919 ts_tablet_manager.cc:1918] T f738929269a54873afa9d51b5ed51ebb P 6f5a5fd7474244c6bfb857386b7e9e0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:34:06.877763 7919 log.cc:1196] T f738929269a54873afa9d51b5ed51ebb P 6f5a5fd7474244c6bfb857386b7e9e0e: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/f738929269a54873afa9d51b5ed51ebb
I20260619 04:34:06.878079 7919 ts_tablet_manager.cc:1939] T f738929269a54873afa9d51b5ed51ebb P 6f5a5fd7474244c6bfb857386b7e9e0e: Deleting consensus metadata
I20260619 04:34:06.879056 7316 catalog_manager.cc:4841] TS 6f5a5fd7474244c6bfb857386b7e9e0e (127.0.97.2:37011): tablet f738929269a54873afa9d51b5ed51ebb (table test-table [id=2048dafd5d144fd4924cc8493f665018]) successfully deleted
I20260619 04:34:06.887437 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:06.863992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52784 (local address 127.0.97.1:38759)
0619 04:34:06.864154 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:06.864160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:06.873525 (+ 9365us) server_negotiation.cc:407] Connection header received
0619 04:34:06.873652 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:06.873658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:06.873732 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:06.873836 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:06.875439 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.876701 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.878373 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:06.878963 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:06.879085 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:06.882098 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:06.882110 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:06.882128 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:06.882162 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:06.884986 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:06.885719 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:06.885725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:06.885727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:06.885871 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:06.886202 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:06.886205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:06.886207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:06.886516 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:06.886743 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:06.887001 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 04:34:06.887144 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 04:34:06.888098 7505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:06.986845 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 7304
I20260619 04:34:07.001706 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 7521
I20260619 04:34:07.013852 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:07.002371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57060 (local address 127.0.97.2:37011)
0619 04:34:07.002524 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:07.002529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:07.002590 (+ 61us) server_negotiation.cc:407] Connection header received
0619 04:34:07.002739 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:07.002743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:07.002814 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:07.002895 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:07.004051 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:07.004860 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:07.006130 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:07.006503 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:07.006614 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:07.008998 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:07.009012 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:07.009029 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:07.009061 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:07.011618 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:07.012217 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:07.012224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:07.012226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:07.012341 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:07.012689 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:07.012692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:07.012694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:07.013038 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:07.013275 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:07.013427 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 04:34:07.013598 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 04:34:07.014480 7650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:07.129750 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 7521
I20260619 04:34:07.147156 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 7668
I20260619 04:34:07.160950 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:07.147794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51984 (local address 127.0.97.3:42411)
0619 04:34:07.147975 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:07.147980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:07.148013 (+ 33us) server_negotiation.cc:407] Connection header received
0619 04:34:07.148184 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:07.148188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:07.148257 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:07.148338 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:07.149582 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:07.150446 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:07.151624 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:07.152247 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:07.152359 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:07.154850 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:07.154869 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:07.154889 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:07.154926 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:07.158491 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:07.159067 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:07.159077 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:07.159081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:07.159212 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:07.159510 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:07.159515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:07.159519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:07.159933 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:07.160206 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:07.160372 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 04:34:07.160604 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 04:34:07.161566 7791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:07.253072 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 7668
I20260619 04:34:07.266367 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 7006
I20260619 04:34:07.267352 7172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:07.279918 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:07.279239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.0.1:58514)
0619 04:34:07.279660 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:07.279760 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":217,"threads_started":1}
I20260619 04:34:07.281179 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:07.280873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42529 (local address 127.0.0.1:58530)
0619 04:34:07.281004 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:07.281064 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 04:34:07.379923 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 7006
I20260619 04:34:07.396077 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 7108
I20260619 04:34:07.396798 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:07.396380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39847 (local address 127.0.0.1:43440)
0619 04:34:07.396582 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:07.396656 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 04:34:07.396973 7319 consensus_peers.cc:487] T 00000000000000000000000000000000 P 974acf9a4cd54e84b8fe000bf11137a8 -> Peer 7a74506f385f403d9104e5dcdb2c851a (127.0.97.61:39847): Couldn't send request to peer 7a74506f385f403d9104e5dcdb2c851a. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39847: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 04:34:07.397114 7362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:07.513123 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 7108
2026-06-19T04:34:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8938 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:34:07 dist-test-slave-n3tq krb5kdc[7930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:34:07 dist-test-slave-n3tq krb5kdc[7930](info): set up 2 sockets
Jun 19 04:34:07 dist-test-slave-n3tq krb5kdc[7930](info): commencing operation
krb5kdc: starting...
W20260619 04:34:08.512962 388 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.008s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:34:08 dist-test-slave-n3tq krb5kdc[7930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:34:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:34:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260619 04:34:08.707990 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38229
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:34:09.018250 7943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:09.018536 7943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:09.018659 7943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:09.025945 7943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:34:09.026055 7943 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 04:34:09.026098 7943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:09.026181 7943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:09.026247 7943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:34:09.026333 7943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:34:09.036031 7943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38229
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:09.037317 7943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:09.038580 7943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:09.049398 7951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:09.049471 7948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:09.049450 7949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:09.050263 7943 server_base.cc:1034] running on GCE node
I20260619 04:34:09.050940 7943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:09.052278 7943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:09.053489 7943 hybrid_clock.cc:648] HybridClock initialized: now 1781843649053416 us; error 51 us; skew 500 ppm
I20260619 04:34:09.053802 7943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:09 dist-test-slave-n3tq krb5kdc[7930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843649, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:09.061995 7943 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260619 04:34:09.063450 7943 webserver.cc:460] Webserver started at http://127.0.97.62:46383/ using document root <none> and password file <none>
I20260619 04:34:09.064138 7943 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:09.064237 7943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:09.064591 7943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:09.067480 7943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "8a3b8f71c5ca4979b1aba4f0881f9a74"
format_stamp: "Formatted at 2026-06-19 04:34:09 on dist-test-slave-n3tq"
server_key: "9cb89141a8ccab35d9017d700b9590ca"
server_key_iv: "797cdba5a3d2c0dfec79c6990462c955"
server_key_version: "encryptionkey@0"
I20260619 04:34:09.068226 7943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "8a3b8f71c5ca4979b1aba4f0881f9a74"
format_stamp: "Formatted at 2026-06-19 04:34:09 on dist-test-slave-n3tq"
server_key: "9cb89141a8ccab35d9017d700b9590ca"
server_key_iv: "797cdba5a3d2c0dfec79c6990462c955"
server_key_version: "encryptionkey@0"
I20260619 04:34:09.073297 7943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 04:34:09.076753 7958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:09.078051 7943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:34:09.078203 7943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "8a3b8f71c5ca4979b1aba4f0881f9a74"
format_stamp: "Formatted at 2026-06-19 04:34:09 on dist-test-slave-n3tq"
server_key: "9cb89141a8ccab35d9017d700b9590ca"
server_key_iv: "797cdba5a3d2c0dfec79c6990462c955"
server_key_version: "encryptionkey@0"
I20260619 04:34:09.078404 7943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:09.111562 7943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:09.115653 7943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:09.115895 7943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:09.135167 7943 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38229
I20260619 04:34:09.135149 8019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38229 every 8 connection(s)
I20260619 04:34:09.136497 7943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:34:09.140069 8020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:09.146927 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 7943
I20260619 04:34:09.147161 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:34:09.147881 388 external_mini_cluster.cc:1428] Setting key b692bb6b82e6811ff32b575a21bfbae0
I20260619 04:34:09.150590 8020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74: Bootstrap starting.
I20260619 04:34:09.153622 8020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:09.154537 8020 log.cc:824] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74: Log is configured to *not* fsync() on all Append() calls
I20260619 04:34:09.157096 8020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74: No bootstrap required, opened a new log
Jun 19 04:34:09 dist-test-slave-n3tq krb5kdc[7930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 04:34:09.164978 8020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3b8f71c5ca4979b1aba4f0881f9a74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38229 } }
I20260619 04:34:09.165411 8020 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:09.165509 8020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3b8f71c5ca4979b1aba4f0881f9a74, State: Initialized, Role: FOLLOWER
I20260619 04:34:09.166098 8020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3b8f71c5ca4979b1aba4f0881f9a74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38229 } }
I20260619 04:34:09.166363 8020 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:34:09.166499 8020 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:34:09.166692 8020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:09.169008 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:09.149886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58980 (local address 127.0.97.62:38229)
0619 04:34:09.150648 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:09.150663 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:09.150713 (+ 50us) server_negotiation.cc:407] Connection header received
0619 04:34:09.152509 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:09.152532 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:09.152947 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:09.153277 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:09.154579 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:09.155913 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:09.157153 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:09.157872 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:09.158045 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:09.162026 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:09.162033 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:09.162050 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:09.162083 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:09.164892 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:09.165675 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:09.165682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:09.165688 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:09.165888 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:09.166276 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:09.166279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:09.166281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:09.166903 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:09.167106 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:09.167551 (+ 445us) server_negotiation.cc:299] Negotiation successful
0619 04:34:09.167899 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":342,"threads_started":1}
I20260619 04:34:09.169227 8020 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3b8f71c5ca4979b1aba4f0881f9a74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38229 } }
I20260619 04:34:09.169720 8020 leader_election.cc:302] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a3b8f71c5ca4979b1aba4f0881f9a74; no voters:
I20260619 04:34:09.170274 8020 leader_election.cc:288] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:34:09.170390 8025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:34:09.171591 8025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74 [term 1 LEADER]: Becoming Leader. State: Replica: 8a3b8f71c5ca4979b1aba4f0881f9a74, State: Running, Role: LEADER
I20260619 04:34:09.172185 8025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3b8f71c5ca4979b1aba4f0881f9a74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38229 } }
I20260619 04:34:09.172405 8020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:34:09.180572 8026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a3b8f71c5ca4979b1aba4f0881f9a74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3b8f71c5ca4979b1aba4f0881f9a74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38229 } } }
I20260619 04:34:09.180996 8026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74 [sys.catalog]: This master's current role is: LEADER
I20260619 04:34:09.183019 8027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3b8f71c5ca4979b1aba4f0881f9a74. Latest consensus state: current_term: 1 leader_uuid: "8a3b8f71c5ca4979b1aba4f0881f9a74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3b8f71c5ca4979b1aba4f0881f9a74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38229 } } }
I20260619 04:34:09.183295 8027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74 [sys.catalog]: This master's current role is: LEADER
W20260619 04:34:09.187299 8041 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 04:34:09.187397 8041 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 04:34:09.187497 8040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:34:09.189632 8040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:34:09.197862 8040 catalog_manager.cc:1347] Generated new cluster ID: 0d5f0c70ba134c2dba03f76bf39ead93
I20260619 04:34:09.197975 8040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:34:09.223096 8040 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:34:09.224386 8040 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:34:09.230369 8040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74: Generated new TSK 0
I20260619 04:34:09.231108 8040 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260619 04:34:09.276259 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38229
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:34:09.595534 8046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:09.595798 8046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:09.595919 8046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:09.603075 8046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:09.603188 8046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:09.603300 8046 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 04:34:09.603403 8046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:34:09.613099 8046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:09.614434 8046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:09.615733 8046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:09.628069 8052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:09.628072 8054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:09.628197 8051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:09.628778 8046 server_base.cc:1034] running on GCE node
I20260619 04:34:09.629341 8046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:09.630198 8046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:09.631395 8046 hybrid_clock.cc:648] HybridClock initialized: now 1781843649631324 us; error 50 us; skew 500 ppm
I20260619 04:34:09.631681 8046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:09 dist-test-slave-n3tq krb5kdc[7930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843649, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:09.634688 8046 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 04:34:09.635972 8046 webserver.cc:460] Webserver started at http://127.0.97.1:36839/ using document root <none> and password file <none>
I20260619 04:34:09.636670 8046 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:09.636756 8046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:09.637012 8046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:09.639741 8046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "eeafa8fcfab041fb8fb588988c4aa694"
format_stamp: "Formatted at 2026-06-19 04:34:09 on dist-test-slave-n3tq"
server_key: "6e4b3738456d9109a7441e1af39987aa"
server_key_iv: "95d79902e807ecaadbdefacce1c89c7d"
server_key_version: "encryptionkey@0"
I20260619 04:34:09.640502 8046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "eeafa8fcfab041fb8fb588988c4aa694"
format_stamp: "Formatted at 2026-06-19 04:34:09 on dist-test-slave-n3tq"
server_key: "6e4b3738456d9109a7441e1af39987aa"
server_key_iv: "95d79902e807ecaadbdefacce1c89c7d"
server_key_version: "encryptionkey@0"
I20260619 04:34:09.645368 8046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 04:34:09.648634 8061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:09.649914 8046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:34:09.650035 8046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "eeafa8fcfab041fb8fb588988c4aa694"
format_stamp: "Formatted at 2026-06-19 04:34:09 on dist-test-slave-n3tq"
server_key: "6e4b3738456d9109a7441e1af39987aa"
server_key_iv: "95d79902e807ecaadbdefacce1c89c7d"
server_key_version: "encryptionkey@0"
I20260619 04:34:09.650206 8046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:09.687595 8046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:09.691637 8046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:09.691849 8046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:09.694249 8046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:09.694334 8046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:09.694453 8046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:09.694507 8046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 04:34:09 dist-test-slave-n3tq krb5kdc[7930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843649, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 04:34:09.716800 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:09.694926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47943 (local address 127.0.97.62:38229)
0619 04:34:09.695415 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:09.695421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:09.696684 (+ 1263us) server_negotiation.cc:407] Connection header received
0619 04:34:09.698813 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:09.698820 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:09.698913 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:09.699041 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:09.701142 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:09.702103 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:09.704437 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:09.705074 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:09.705210 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:09.710063 (+ 4853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:09.710068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:09.710071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:09.710105 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:09.713277 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:09.714260 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:09.714266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:09.714270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:09.714423 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:09.714979 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:09.714985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:09.714989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:09.715274 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:09.715402 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:09.716186 (+ 784us) server_negotiation.cc:299] Negotiation successful
0619 04:34:09.716403 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":261,"threads_started":1}
I20260619 04:34:09.718211 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:09.695645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38229 (local address 127.0.97.1:47943)
0619 04:34:09.696490 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:09.696559 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:09.698431 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:09.699352 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:09.699367 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:09.700084 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:09.700948 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:09.700982 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:09.702308 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:09.702313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:09.704220 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:09.704236 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:09.705313 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:09.705321 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:09.705612 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:09.706397 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:09.706442 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:09.709794 (+ 3352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:09.713432 (+ 3638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:09.713443 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:09.713468 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:09.714091 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:09.714561 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:09.714566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:09.714570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:09.714860 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:09.715409 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:09.715419 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:09.715831 (+ 412us) client_negotiation.cc:769] Sending connection context
0619 04:34:09.716073 (+ 242us) client_negotiation.cc:240] Negotiation successful
0619 04:34:09.716656 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":332,"threads_started":1}
I20260619 04:34:09.736678 8046 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35349
I20260619 04:34:09.736680 8181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35349 every 8 connection(s)
I20260619 04:34:09.738083 8046 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:34:09.739995 8182 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38229
I20260619 04:34:09.740885 8182 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:09.741676 8182 heartbeater.cc:511] Master 127.0.97.62:38229 requested a full tablet report, sending...
I20260619 04:34:09.742720 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:09.728763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40671 (local address 127.0.97.62:38229)
0619 04:34:09.728963 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:09.728970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:09.729653 (+ 683us) server_negotiation.cc:407] Connection header received
0619 04:34:09.729810 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:09.729816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:09.729900 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:09.730039 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:09.731300 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:09.732415 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:09.734658 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:09.735053 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:09.735186 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:09.737445 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:09.737449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:09.737453 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:09.737481 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:09.740216 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:09.740930 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:09.740935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:09.740938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:09.741086 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:09.741417 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:09.741421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:09.741424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:09.741764 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:09.741906 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:09.742243 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 04:34:09.742450 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 04:34:09.742758 8160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:09.728819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38229 (local address 127.0.97.1:40671)
0619 04:34:09.729538 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:09.729558 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:09.729699 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:09.730069 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:09.730072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:09.730690 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:09.731139 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:09.731150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:09.732625 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:09.732632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:09.734499 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:09.734513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:09.735280 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:09.735286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:09.735460 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:09.736198 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:09.736224 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:09.737285 (+ 1061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:09.740316 (+ 3031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:09.740320 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:09.740323 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:09.740821 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:09.741186 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:09.741190 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:09.741192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:09.741318 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:09.741867 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:09.741872 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:09.742020 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 04:34:09.742126 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 04:34:09.742372 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":438,"threads_started":1}
I20260619 04:34:09.743871 7975 ts_manager.cc:194] Registered new tserver with Master: eeafa8fcfab041fb8fb588988c4aa694 (127.0.97.1:35349)
I20260619 04:34:09.745296 7975 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47943
I20260619 04:34:09.746379 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 8046
I20260619 04:34:09.746582 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:34:09.747382 388 external_mini_cluster.cc:1428] Setting key 44611d126f47bb238d6e3430d9b3ad80
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260619 04:34:09.786783 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38229
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:34:10.099548 8186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:10.099844 8186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:10.099967 8186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:10.107298 8186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:10.107429 8186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:10.107549 8186 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 04:34:10.107656 8186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:34:10.117547 8186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:10.118844 8186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:10.120324 8186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:10.132736 8192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:10.132776 8194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:10.132938 8191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:10.133536 8186 server_base.cc:1034] running on GCE node
I20260619 04:34:10.134167 8186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:10.135243 8186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:10.136523 8186 hybrid_clock.cc:648] HybridClock initialized: now 1781843650136418 us; error 58 us; skew 500 ppm
I20260619 04:34:10.136850 8186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:10 dist-test-slave-n3tq krb5kdc[7930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843650, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:10.140396 8186 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 04:34:10.141971 8186 webserver.cc:460] Webserver started at http://127.0.97.2:42409/ using document root <none> and password file <none>
I20260619 04:34:10.142788 8186 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:10.142901 8186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:10.143273 8186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:10.146960 8186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "31b8fdefbb5840d78f3dfb53fb70973a"
format_stamp: "Formatted at 2026-06-19 04:34:10 on dist-test-slave-n3tq"
server_key: "a73b5eb7593ef0c8346022c9c9db09fd"
server_key_iv: "5fe00c58c7019318e416ab02bcfc049f"
server_key_version: "encryptionkey@0"
I20260619 04:34:10.147895 8186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "31b8fdefbb5840d78f3dfb53fb70973a"
format_stamp: "Formatted at 2026-06-19 04:34:10 on dist-test-slave-n3tq"
server_key: "a73b5eb7593ef0c8346022c9c9db09fd"
server_key_iv: "5fe00c58c7019318e416ab02bcfc049f"
server_key_version: "encryptionkey@0"
I20260619 04:34:10.154089 8186 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 04:34:10.157437 8201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:10.158773 8186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:34:10.158941 8186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "31b8fdefbb5840d78f3dfb53fb70973a"
format_stamp: "Formatted at 2026-06-19 04:34:10 on dist-test-slave-n3tq"
server_key: "a73b5eb7593ef0c8346022c9c9db09fd"
server_key_iv: "5fe00c58c7019318e416ab02bcfc049f"
server_key_version: "encryptionkey@0"
I20260619 04:34:10.159148 8186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:10.181181 8186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:10.184947 8186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:10.185184 8186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:10.187641 8186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:10.187774 8186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:10.187906 8186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:10.187970 8186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 04:34:10 dist-test-slave-n3tq krb5kdc[7930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843650, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 04:34:10.210695 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:10.188414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32907 (local address 127.0.97.62:38229)
0619 04:34:10.188641 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:10.188645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:10.190056 (+ 1411us) server_negotiation.cc:407] Connection header received
0619 04:34:10.192317 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:10.192324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:10.192407 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:10.192523 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:10.194923 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.195942 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.198737 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.199226 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.199352 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:10.204200 (+ 4848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:10.204207 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:10.204212 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:10.204254 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:10.207539 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:10.208350 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:10.208357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:10.208361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:10.208533 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:10.208928 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:10.208933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:10.208937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:10.209305 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:10.209440 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:10.210070 (+ 630us) server_negotiation.cc:299] Negotiation successful
0619 04:34:10.210322 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260619 04:34:10.211431 8210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:10.189136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38229 (local address 127.0.97.2:32907)
0619 04:34:10.189917 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:10.189953 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:10.191955 (+ 2002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:10.193024 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:10.193039 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:10.193848 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:10.194690 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:10.194720 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.196114 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.196121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:10.198583 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:10.198599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.199541 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.199554 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:10.199905 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:10.200698 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:10.200738 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:10.203963 (+ 3225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:10.207689 (+ 3726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:10.207698 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:10.207716 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:10.208210 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:10.208636 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:10.208639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:10.208641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:10.208828 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:10.209410 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:10.209415 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:10.209723 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 04:34:10.209948 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 04:34:10.210314 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":286,"threads_started":1}
I20260619 04:34:10.230581 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:10.216446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48523 (local address 127.0.97.62:38229)
0619 04:34:10.216659 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:10.216665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:10.217428 (+ 763us) server_negotiation.cc:407] Connection header received
0619 04:34:10.217579 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:10.217584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:10.217652 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:10.217745 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:10.219303 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.220182 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.222231 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.222768 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.222877 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:10.225171 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:10.225178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:10.225182 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:10.225218 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:10.227748 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:10.228614 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:10.228620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:10.228623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:10.228757 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:10.229184 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:10.229187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:10.229189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:10.229534 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:10.229667 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:10.230046 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 04:34:10.230233 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 04:34:10.230654 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:10.216688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38229 (local address 127.0.97.2:48523)
0619 04:34:10.217333 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:10.217348 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:10.217474 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:10.217800 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:10.217805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:10.218567 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:10.219065 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:10.219080 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.220319 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.220325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:10.222077 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:10.222090 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.222950 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.222957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:10.223131 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:10.223783 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:10.223807 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:10.224979 (+ 1172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:10.227863 (+ 2884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:10.227868 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:10.227873 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:10.228440 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:10.228904 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:10.228909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:10.228916 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:10.229073 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:10.229642 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:10.229647 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:10.229818 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 04:34:10.229927 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 04:34:10.230216 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":416,"threads_started":1}
I20260619 04:34:10.236012 8320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36779 every 8 connection(s)
I20260619 04:34:10.236008 8186 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36779
I20260619 04:34:10.237586 8186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:34:10.239378 8321 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38229
I20260619 04:34:10.239578 8321 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:10.240139 8321 heartbeater.cc:511] Master 127.0.97.62:38229 requested a full tablet report, sending...
I20260619 04:34:10.241381 7975 ts_manager.cc:194] Registered new tserver with Master: 31b8fdefbb5840d78f3dfb53fb70973a (127.0.97.2:36779)
I20260619 04:34:10.242144 7975 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:32907
I20260619 04:34:10.245841 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 8186
I20260619 04:34:10.246001 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:34:10.246476 388 external_mini_cluster.cc:1428] Setting key 8d11749d7314dae21e4a08e3e3f123d7
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260619 04:34:10.288172 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38229
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:34:10.595129 8325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:10.595394 8325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:10.595485 8325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:10.602417 8325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:10.602523 8325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:10.602612 8325 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 04:34:10.602689 8325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:34:10.615094 8325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:10.616286 8325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:10.617576 8325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:10.629645 8331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:10.629571 8333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:10.629606 8330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:10.630422 8325 server_base.cc:1034] running on GCE node
I20260619 04:34:10.630961 8325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:10.631762 8325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:10.632961 8325 hybrid_clock.cc:648] HybridClock initialized: now 1781843650632890 us; error 40 us; skew 500 ppm
I20260619 04:34:10.633213 8325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:10 dist-test-slave-n3tq krb5kdc[7930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843650, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:10.635929 8325 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 04:34:10.637245 8325 webserver.cc:460] Webserver started at http://127.0.97.3:35605/ using document root <none> and password file <none>
I20260619 04:34:10.637882 8325 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:10.637974 8325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:10.638249 8325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:10.640936 8325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "b72019478929468888fcd697a4d4968a"
format_stamp: "Formatted at 2026-06-19 04:34:10 on dist-test-slave-n3tq"
server_key: "e15e8a7faf2e4ad90c355a47f22df48c"
server_key_iv: "dbd9c0717166aff721f2a7ab32f28b58"
server_key_version: "encryptionkey@0"
I20260619 04:34:10.641644 8325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b72019478929468888fcd697a4d4968a"
format_stamp: "Formatted at 2026-06-19 04:34:10 on dist-test-slave-n3tq"
server_key: "e15e8a7faf2e4ad90c355a47f22df48c"
server_key_iv: "dbd9c0717166aff721f2a7ab32f28b58"
server_key_version: "encryptionkey@0"
I20260619 04:34:10.646256 8325 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 04:34:10.649437 8340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:10.650596 8325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 04:34:10.650740 8325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "b72019478929468888fcd697a4d4968a"
format_stamp: "Formatted at 2026-06-19 04:34:10 on dist-test-slave-n3tq"
server_key: "e15e8a7faf2e4ad90c355a47f22df48c"
server_key_iv: "dbd9c0717166aff721f2a7ab32f28b58"
server_key_version: "encryptionkey@0"
I20260619 04:34:10.650918 8325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:10.674715 8325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:10.678354 8325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:10.678586 8325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:10.681071 8325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:10.681156 8325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:10.681273 8325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:10.681329 8325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 04:34:10 dist-test-slave-n3tq krb5kdc[7930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843650, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 04:34:10.703980 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:10.681545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60041 (local address 127.0.97.62:38229)
0619 04:34:10.681735 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:10.681741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:10.683250 (+ 1509us) server_negotiation.cc:407] Connection header received
0619 04:34:10.686088 (+ 2838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:10.686095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:10.686215 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:10.686291 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:10.688880 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.689596 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.692324 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.692859 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.692996 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:10.697805 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:10.697813 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:10.697818 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:10.697859 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:10.700819 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:10.701698 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:10.701704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:10.701707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:10.701867 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:10.702337 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:10.702342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:10.702346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:10.702612 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:10.702740 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:10.703425 (+ 685us) server_negotiation.cc:299] Negotiation successful
0619 04:34:10.703637 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 04:34:10.705181 8349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:10.682217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38229 (local address 127.0.97.3:60041)
0619 04:34:10.683020 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:10.683068 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:10.685726 (+ 2658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:10.686856 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:10.686874 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:10.687718 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:10.688672 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:10.688710 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.689760 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.689766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:10.692149 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:10.692165 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.693117 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.693124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:10.693446 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:10.694173 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:10.694227 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:10.697479 (+ 3252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:10.700952 (+ 3473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:10.700960 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:10.700981 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:10.701556 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:10.702006 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:10.702010 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:10.702014 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:10.702211 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:10.702733 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:10.702740 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:10.703079 (+ 339us) client_negotiation.cc:769] Sending connection context
0619 04:34:10.703314 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 04:34:10.703819 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":278,"threads_started":1}
I20260619 04:34:10.724639 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:10.710815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59589 (local address 127.0.97.62:38229)
0619 04:34:10.710984 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:10.710991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:10.711583 (+ 592us) server_negotiation.cc:407] Connection header received
0619 04:34:10.711747 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:10.711750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:10.711833 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:10.711936 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:10.713125 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.713916 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.715980 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.716488 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.716629 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:10.718736 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:10.718741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:10.718744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:10.718771 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:10.721838 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:10.722738 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:10.722743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:10.722746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:10.722901 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:10.723253 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:10.723258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:10.723261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:10.723501 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:10.723627 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:10.724060 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 04:34:10.724265 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 04:34:10.724932 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:10.710832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38229 (local address 127.0.97.3:59589)
0619 04:34:10.711469 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:10.711491 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:10.711646 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:10.711966 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:10.711971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:10.712580 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:10.712976 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:10.712988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.714049 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.714054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:10.715714 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:10.715731 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.716684 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.716692 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:10.716890 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:10.717492 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:10.717515 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:10.718558 (+ 1043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:10.721996 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:10.722001 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:10.722005 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:10.722620 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:10.723037 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:10.723040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:10.723043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:10.723154 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:10.723609 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:10.723613 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:10.723802 (+ 189us) client_negotiation.cc:769] Sending connection context
0619 04:34:10.723930 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 04:34:10.724253 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":331,"threads_started":1}
I20260619 04:34:10.732270 8325 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38111
I20260619 04:34:10.732287 8459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38111 every 8 connection(s)
I20260619 04:34:10.733428 8325 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:34:10.735188 8460 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38229
I20260619 04:34:10.735380 8460 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:10.736023 8460 heartbeater.cc:511] Master 127.0.97.62:38229 requested a full tablet report, sending...
I20260619 04:34:10.736492 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 8325
I20260619 04:34:10.736689 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:34:10.737176 388 external_mini_cluster.cc:1428] Setting key cb74a055850460f3261f706dd807dea6
I20260619 04:34:10.738338 7975 ts_manager.cc:194] Registered new tserver with Master: b72019478929468888fcd697a4d4968a (127.0.97.3:38111)
I20260619 04:34:10.739265 7975 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60041
I20260619 04:34:10.740212 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:34:10.748389 8182 heartbeater.cc:503] Master 127.0.97.62:38229 was elected leader, sending a full tablet report...
Jun 19 04:34:10 dist-test-slave-n3tq krb5kdc[7930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843650, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 04:34:10 dist-test-slave-n3tq krb5kdc[7930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:10 dist-test-slave-n3tq krb5kdc[7930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 04:34:10.784297 8070 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:10.774854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.0.97.62:38229)
0619 04:34:10.775008 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:10.775013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:10.775216 (+ 203us) server_negotiation.cc:407] Connection header received
0619 04:34:10.775362 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:10.775366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:10.775429 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:10.775520 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:10.776766 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.777468 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.778656 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.779047 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.779135 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:10.784094 (+ 4959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58986: BlockingRecv error: recv got EOF from 127.0.0.1:58986 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 04:34:10 dist-test-slave-n3tq krb5kdc[7930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843650, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 04:34:10.798733 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:10.787016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58988 (local address 127.0.97.62:38229)
0619 04:34:10.787201 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:10.787207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:10.787320 (+ 113us) server_negotiation.cc:407] Connection header received
0619 04:34:10.787474 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:10.787480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:10.787554 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:10.787634 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:10.788735 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.789416 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.790557 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.790955 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.791059 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:10.794248 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:10.794253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:10.794256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:10.794285 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:10.796496 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:10.797078 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:10.797084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:10.797088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:10.797205 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:10.797518 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:10.797523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:10.797526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:10.797764 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:10.798007 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:10.798250 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 04:34:10.798437 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 04:34:10.802299 7974 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:34:10.808995 7974 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:34:10.842695 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:10.821127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59324 (local address 127.0.97.2:36779)
0619 04:34:10.830769 (+ 9642us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:10.830778 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:10.835333 (+ 4555us) server_negotiation.cc:407] Connection header received
0619 04:34:10.835435 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:10.835442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:10.835675 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:10.835843 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:10.836620 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.837857 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.840089 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.841227 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.841355 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:10.841595 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:10.842228 (+ 633us) server_negotiation.cc:299] Negotiation successful
0619 04:34:10.842408 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9480,"thread_start_us":9349,"threads_started":1}
I20260619 04:34:10.843101 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:10.820434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36779 (local address 127.0.0.1:59324)
0619 04:34:10.835052 (+ 14618us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:10.835073 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:10.835196 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:10.836005 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:10.836009 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:10.836032 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:10.836408 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:10.836417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.838004 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.838008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:10.839938 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:10.839956 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.841384 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.841388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:10.841511 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:10.841543 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 04:34:10.841671 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 04:34:10.841857 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14510,"mutex_wait_us":61,"thread_start_us":14005,"threads_started":1}
I20260619 04:34:10.843230 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:10.820052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38111 (local address 127.0.0.1:40630)
0619 04:34:10.834737 (+ 14685us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:10.834760 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:10.834876 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:10.835838 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:10.835843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:10.835865 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:10.836336 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:10.836351 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.837987 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.837992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:10.840378 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:10.840392 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.842369 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.842375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:10.842531 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:10.842563 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 04:34:10.842638 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 04:34:10.842762 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14550,"mutex_wait_us":2,"thread_start_us":14747,"threads_started":1}
I20260619 04:34:10.843813 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:10.820695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40630 (local address 127.0.97.3:38111)
0619 04:34:10.830338 (+ 9643us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:10.830349 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:10.835013 (+ 4664us) server_negotiation.cc:407] Connection header received
0619 04:34:10.835121 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:10.835127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:10.835365 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:10.835549 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:10.836600 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.837830 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.840552 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.841889 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.842000 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:10.842234 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:10.843070 (+ 836us) server_negotiation.cc:299] Negotiation successful
0619 04:34:10.843238 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9435,"thread_start_us":9232,"threads_started":1}
I20260619 04:34:10.844240 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:10.819924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35349 (local address 127.0.0.1:55310)
0619 04:34:10.834030 (+ 14106us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:10.834080 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:10.834287 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:10.835117 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:10.835124 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:10.835153 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:10.835561 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:10.835586 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.837572 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.837577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:10.840386 (+ 2809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:10.840394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.843477 (+ 3083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.843483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:10.843659 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:10.843689 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 04:34:10.843766 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 04:34:10.843888 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13860,"mutex_wait_us":50,"spinlock_wait_cycles":9472,"thread_start_us":13722,"threads_started":1}
I20260619 04:34:10.844241 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:10.820733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55310 (local address 127.0.97.1:35349)
0619 04:34:10.833584 (+ 12851us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:10.833593 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:10.834499 (+ 906us) server_negotiation.cc:407] Connection header received
0619 04:34:10.834627 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:10.834634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:10.834897 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:10.835766 (+ 869us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:10.835944 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.837447 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.841687 (+ 4240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:10.843037 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:10.843154 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:10.843335 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:10.843816 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 04:34:10.843964 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12700,"thread_start_us":12564,"threads_started":1}
I20260619 04:34:10.846956 8390 tablet_service.cc:1467] Processing CreateTablet for tablet 319f290b8a8f40c28f79ee509f5aae20 (DEFAULT_TABLE table=test-table [id=a7abc34ae59c4730b92b656049f62fae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:34:10.847064 8251 tablet_service.cc:1467] Processing CreateTablet for tablet 319f290b8a8f40c28f79ee509f5aae20 (DEFAULT_TABLE table=test-table [id=a7abc34ae59c4730b92b656049f62fae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:34:10.847527 8112 tablet_service.cc:1467] Processing CreateTablet for tablet 319f290b8a8f40c28f79ee509f5aae20 (DEFAULT_TABLE table=test-table [id=a7abc34ae59c4730b92b656049f62fae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:34:10.848208 8390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 319f290b8a8f40c28f79ee509f5aae20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:10.848605 8251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 319f290b8a8f40c28f79ee509f5aae20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:10.848704 8112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 319f290b8a8f40c28f79ee509f5aae20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:10.859272 8484 tablet_bootstrap.cc:492] T 319f290b8a8f40c28f79ee509f5aae20 P b72019478929468888fcd697a4d4968a: Bootstrap starting.
I20260619 04:34:10.862263 8484 tablet_bootstrap.cc:654] T 319f290b8a8f40c28f79ee509f5aae20 P b72019478929468888fcd697a4d4968a: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:10.863451 8484 log.cc:824] T 319f290b8a8f40c28f79ee509f5aae20 P b72019478929468888fcd697a4d4968a: Log is configured to *not* fsync() on all Append() calls
I20260619 04:34:10.865726 8486 tablet_bootstrap.cc:492] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a: Bootstrap starting.
I20260619 04:34:10.866339 8484 tablet_bootstrap.cc:492] T 319f290b8a8f40c28f79ee509f5aae20 P b72019478929468888fcd697a4d4968a: No bootstrap required, opened a new log
I20260619 04:34:10.866573 8484 ts_tablet_manager.cc:1397] T 319f290b8a8f40c28f79ee509f5aae20 P b72019478929468888fcd697a4d4968a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 04:34:10.868381 8486 tablet_bootstrap.cc:654] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:10.869355 8486 log.cc:824] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a: Log is configured to *not* fsync() on all Append() calls
I20260619 04:34:10.869714 8485 tablet_bootstrap.cc:492] T 319f290b8a8f40c28f79ee509f5aae20 P eeafa8fcfab041fb8fb588988c4aa694: Bootstrap starting.
I20260619 04:34:10.874125 8485 tablet_bootstrap.cc:654] T 319f290b8a8f40c28f79ee509f5aae20 P eeafa8fcfab041fb8fb588988c4aa694: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:10.874480 8484 raft_consensus.cc:348] T 319f290b8a8f40c28f79ee509f5aae20 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } }
I20260619 04:34:10.874866 8484 raft_consensus.cc:374] T 319f290b8a8f40c28f79ee509f5aae20 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:10.874964 8484 raft_consensus.cc:729] T 319f290b8a8f40c28f79ee509f5aae20 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b72019478929468888fcd697a4d4968a, State: Initialized, Role: FOLLOWER
I20260619 04:34:10.875567 8485 log.cc:824] T 319f290b8a8f40c28f79ee509f5aae20 P eeafa8fcfab041fb8fb588988c4aa694: Log is configured to *not* fsync() on all Append() calls
I20260619 04:34:10.875552 8484 consensus_queue.cc:260] T 319f290b8a8f40c28f79ee509f5aae20 P b72019478929468888fcd697a4d4968a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } }
I20260619 04:34:10.876619 8460 heartbeater.cc:503] Master 127.0.97.62:38229 was elected leader, sending a full tablet report...
I20260619 04:34:10.877942 8484 ts_tablet_manager.cc:1428] T 319f290b8a8f40c28f79ee509f5aae20 P b72019478929468888fcd697a4d4968a: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260619 04:34:10.879091 8486 tablet_bootstrap.cc:492] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a: No bootstrap required, opened a new log
I20260619 04:34:10.879179 8485 tablet_bootstrap.cc:492] T 319f290b8a8f40c28f79ee509f5aae20 P eeafa8fcfab041fb8fb588988c4aa694: No bootstrap required, opened a new log
I20260619 04:34:10.879400 8486 ts_tablet_manager.cc:1397] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260619 04:34:10.879477 8485 ts_tablet_manager.cc:1397] T 319f290b8a8f40c28f79ee509f5aae20 P eeafa8fcfab041fb8fb588988c4aa694: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.006s
I20260619 04:34:10.886832 8485 raft_consensus.cc:348] T 319f290b8a8f40c28f79ee509f5aae20 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } }
I20260619 04:34:10.886812 8486 raft_consensus.cc:348] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } }
I20260619 04:34:10.887130 8486 raft_consensus.cc:374] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:10.887143 8485 raft_consensus.cc:374] T 319f290b8a8f40c28f79ee509f5aae20 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:10.887224 8485 raft_consensus.cc:729] T 319f290b8a8f40c28f79ee509f5aae20 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeafa8fcfab041fb8fb588988c4aa694, State: Initialized, Role: FOLLOWER
I20260619 04:34:10.887219 8486 raft_consensus.cc:729] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b8fdefbb5840d78f3dfb53fb70973a, State: Initialized, Role: FOLLOWER
I20260619 04:34:10.887738 8485 consensus_queue.cc:260] T 319f290b8a8f40c28f79ee509f5aae20 P eeafa8fcfab041fb8fb588988c4aa694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } }
I20260619 04:34:10.887739 8486 consensus_queue.cc:260] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } }
I20260619 04:34:10.889245 8485 ts_tablet_manager.cc:1428] T 319f290b8a8f40c28f79ee509f5aae20 P eeafa8fcfab041fb8fb588988c4aa694: Time spent starting tablet: real 0.010s user 0.004s sys 0.005s
I20260619 04:34:10.890250 8321 heartbeater.cc:503] Master 127.0.97.62:38229 was elected leader, sending a full tablet report...
I20260619 04:34:10.890885 8486 ts_tablet_manager.cc:1428] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
W20260619 04:34:10.984889 8461 tablet.cc:2369] T 319f290b8a8f40c28f79ee509f5aae20 P b72019478929468888fcd697a4d4968a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:34:10.989401 8322 tablet.cc:2369] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:34:10.990658 8183 tablet.cc:2369] T 319f290b8a8f40c28f79ee509f5aae20 P eeafa8fcfab041fb8fb588988c4aa694: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:34:11.059273 8492 raft_consensus.cc:482] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:34:11.059584 8492 raft_consensus.cc:504] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } }
I20260619 04:34:11.061278 8492 leader_election.cc:288] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eeafa8fcfab041fb8fb588988c4aa694 (127.0.97.1:35349), b72019478929468888fcd697a4d4968a (127.0.97.3:38111)
I20260619 04:34:11.067776 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:11.061756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33095 (local address 127.0.97.3:38111)
0619 04:34:11.061940 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:11.061946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:11.062625 (+ 679us) server_negotiation.cc:407] Connection header received
0619 04:34:11.062786 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:11.062790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:11.062847 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:11.062925 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:11.063571 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.064257 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.066004 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.066824 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.066921 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:11.067009 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:11.067409 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 04:34:11.067530 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 04:34:11.067775 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:11.061637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38553 (local address 127.0.97.1:35349)
0619 04:34:11.061833 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:11.061840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:11.062544 (+ 704us) server_negotiation.cc:407] Connection header received
0619 04:34:11.062732 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:11.062737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:11.062814 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:11.062879 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:11.063473 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.064173 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.066146 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.066888 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.067018 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:11.067112 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:11.067424 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 04:34:11.067536 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 04:34:11.067921 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:11.061587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38111 (local address 127.0.97.2:33095)
0619 04:34:11.062511 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:11.062528 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:11.062651 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:11.062950 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:11.062954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:11.062976 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:11.063398 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:11.063409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.064376 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.064380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:11.065858 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:11.065868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.067029 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.067035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:11.067208 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:11.067286 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 04:34:11.067356 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 04:34:11.067473 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":36,"thread_start_us":538,"threads_started":1}
I20260619 04:34:11.068015 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:11.061486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35349 (local address 127.0.97.2:38553)
0619 04:34:11.062431 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:11.062450 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:11.062578 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:11.062920 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:11.062925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:11.062946 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:11.063332 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:11.063343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.064297 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.064302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:11.065997 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:11.066011 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.067114 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.067119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:11.067259 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:11.067305 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 04:34:11.067386 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 04:34:11.067504 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":43,"thread_start_us":588,"threads_started":1}
I20260619 04:34:11.068883 8132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319f290b8a8f40c28f79ee509f5aae20" candidate_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeafa8fcfab041fb8fb588988c4aa694" is_pre_election: true
I20260619 04:34:11.068881 8410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319f290b8a8f40c28f79ee509f5aae20" candidate_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b72019478929468888fcd697a4d4968a" is_pre_election: true
I20260619 04:34:11.069321 8132 raft_consensus.cc:2454] T 319f290b8a8f40c28f79ee509f5aae20 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b8fdefbb5840d78f3dfb53fb70973a in term 0.
I20260619 04:34:11.069321 8410 raft_consensus.cc:2454] T 319f290b8a8f40c28f79ee509f5aae20 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b8fdefbb5840d78f3dfb53fb70973a in term 0.
I20260619 04:34:11.069864 8204 leader_election.cc:302] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b8fdefbb5840d78f3dfb53fb70973a, eeafa8fcfab041fb8fb588988c4aa694; no voters:
I20260619 04:34:11.070220 8492 raft_consensus.cc:2781] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:34:11.070356 8492 raft_consensus.cc:482] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:34:11.070449 8492 raft_consensus.cc:3037] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:11.072541 8492 raft_consensus.cc:504] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } }
I20260619 04:34:11.073134 8492 leader_election.cc:288] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a [CANDIDATE]: Term 1 election: Requested vote from peers eeafa8fcfab041fb8fb588988c4aa694 (127.0.97.1:35349), b72019478929468888fcd697a4d4968a (127.0.97.3:38111)
I20260619 04:34:11.073580 8132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319f290b8a8f40c28f79ee509f5aae20" candidate_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeafa8fcfab041fb8fb588988c4aa694"
I20260619 04:34:11.073640 8410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319f290b8a8f40c28f79ee509f5aae20" candidate_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b72019478929468888fcd697a4d4968a"
I20260619 04:34:11.073755 8132 raft_consensus.cc:3037] T 319f290b8a8f40c28f79ee509f5aae20 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:11.073827 8410 raft_consensus.cc:3037] T 319f290b8a8f40c28f79ee509f5aae20 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:11.076275 8132 raft_consensus.cc:2454] T 319f290b8a8f40c28f79ee509f5aae20 P eeafa8fcfab041fb8fb588988c4aa694 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b8fdefbb5840d78f3dfb53fb70973a in term 1.
I20260619 04:34:11.076514 8410 raft_consensus.cc:2454] T 319f290b8a8f40c28f79ee509f5aae20 P b72019478929468888fcd697a4d4968a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b8fdefbb5840d78f3dfb53fb70973a in term 1.
I20260619 04:34:11.076653 8204 leader_election.cc:302] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b8fdefbb5840d78f3dfb53fb70973a, eeafa8fcfab041fb8fb588988c4aa694; no voters:
I20260619 04:34:11.076931 8492 raft_consensus.cc:2781] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:34:11.077324 8492 raft_consensus.cc:686] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a [term 1 LEADER]: Becoming Leader. State: Replica: 31b8fdefbb5840d78f3dfb53fb70973a, State: Running, Role: LEADER
I20260619 04:34:11.077771 8492 consensus_queue.cc:237] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } }
I20260619 04:34:11.082024 7974 catalog_manager.cc:5495] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a reported cstate change: term changed from 0 to 1, leader changed from <none> to 31b8fdefbb5840d78f3dfb53fb70973a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } health_report { overall_health: HEALTHY } } }
I20260619 04:34:11.144721 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:11.139594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59332 (local address 127.0.97.2:36779)
0619 04:34:11.139949 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:11.139955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:11.139983 (+ 28us) server_negotiation.cc:407] Connection header received
0619 04:34:11.140077 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:11.140080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:11.140156 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:11.140297 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:34:11.140676 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.141373 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.142702 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.143108 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.143211 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:11.143629 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:34:11.144136 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:34:11.144272 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:11.144348 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 04:34:11.144424 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260619 04:34:11.152173 8410 raft_consensus.cc:1261] T 319f290b8a8f40c28f79ee509f5aae20 P b72019478929468888fcd697a4d4968a [term 1 FOLLOWER]: Refusing update from remote peer 31b8fdefbb5840d78f3dfb53fb70973a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:34:11.152231 8132 raft_consensus.cc:1261] T 319f290b8a8f40c28f79ee509f5aae20 P eeafa8fcfab041fb8fb588988c4aa694 [term 1 FOLLOWER]: Refusing update from remote peer 31b8fdefbb5840d78f3dfb53fb70973a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:34:11.153090 8495 consensus_queue.cc:1035] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:34:11.153434 8492 consensus_queue.cc:1035] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:34:11.169833 8499 mvcc.cc:205] Tried to move back new op lower bound from 7298431595110801408 to 7298431594818064384. Current Snapshot: MvccSnapshot[applied={T|T < 7298431595110801408}]
I20260619 04:34:11.177529 7975 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:34:11.177795 7975 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:34:11.181257 7975 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74: Sending DeleteTablet for 3 replicas of tablet 319f290b8a8f40c28f79ee509f5aae20
I20260619 04:34:11.182372 8112 tablet_service.cc:1514] Processing DeleteTablet for tablet 319f290b8a8f40c28f79ee509f5aae20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:34:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55310
I20260619 04:34:11.182483 8251 tablet_service.cc:1514] Processing DeleteTablet for tablet 319f290b8a8f40c28f79ee509f5aae20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:34:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59324
I20260619 04:34:11.183100 8390 tablet_service.cc:1514] Processing DeleteTablet for tablet 319f290b8a8f40c28f79ee509f5aae20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:34:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40630
I20260619 04:34:11.183951 7975 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:34:11.184906 7975 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:34:11.192605 8111 tablet_service.cc:1467] Processing CreateTablet for tablet a88c799fec3640ee98baef97250e84d5 (DEFAULT_TABLE table=test-table [id=28ee7b18577d4a64a918f38fef1021d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:34:11.193187 8111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88c799fec3640ee98baef97250e84d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:11.197566 8389 tablet_service.cc:1467] Processing CreateTablet for tablet a88c799fec3640ee98baef97250e84d5 (DEFAULT_TABLE table=test-table [id=28ee7b18577d4a64a918f38fef1021d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:34:11.198137 8389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88c799fec3640ee98baef97250e84d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:11.201640 8250 tablet_service.cc:1467] Processing CreateTablet for tablet a88c799fec3640ee98baef97250e84d5 (DEFAULT_TABLE table=test-table [id=28ee7b18577d4a64a918f38fef1021d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:34:11.202234 8250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88c799fec3640ee98baef97250e84d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:11.205986 8510 tablet_replica.cc:330] T 319f290b8a8f40c28f79ee509f5aae20 P b72019478929468888fcd697a4d4968a: stopping tablet replica
I20260619 04:34:11.206538 8510 raft_consensus.cc:2229] T 319f290b8a8f40c28f79ee509f5aae20 P b72019478929468888fcd697a4d4968a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:34:11.207257 8510 raft_consensus.cc:2258] T 319f290b8a8f40c28f79ee509f5aae20 P b72019478929468888fcd697a4d4968a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:34:11.208127 8508 tablet_replica.cc:330] T 319f290b8a8f40c28f79ee509f5aae20 P eeafa8fcfab041fb8fb588988c4aa694: stopping tablet replica
I20260619 04:34:11.210448 8510 ts_tablet_manager.cc:1905] T 319f290b8a8f40c28f79ee509f5aae20 P b72019478929468888fcd697a4d4968a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:11.211072 8508 raft_consensus.cc:2229] T 319f290b8a8f40c28f79ee509f5aae20 P eeafa8fcfab041fb8fb588988c4aa694 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:34:11.211720 8508 raft_consensus.cc:2258] T 319f290b8a8f40c28f79ee509f5aae20 P eeafa8fcfab041fb8fb588988c4aa694 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:34:11.210919 8484 tablet_bootstrap.cc:492] T a88c799fec3640ee98baef97250e84d5 P b72019478929468888fcd697a4d4968a: Bootstrap starting.
I20260619 04:34:11.212365 8485 tablet_bootstrap.cc:492] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694: Bootstrap starting.
I20260619 04:34:11.213891 8484 tablet_bootstrap.cc:654] T a88c799fec3640ee98baef97250e84d5 P b72019478929468888fcd697a4d4968a: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:11.213924 8485 tablet_bootstrap.cc:654] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:11.217190 8485 tablet_bootstrap.cc:492] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694: No bootstrap required, opened a new log
I20260619 04:34:11.217415 8485 ts_tablet_manager.cc:1397] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 04:34:11.218225 8485 raft_consensus.cc:348] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.218493 8485 raft_consensus.cc:374] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:11.218631 8485 raft_consensus.cc:729] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeafa8fcfab041fb8fb588988c4aa694, State: Initialized, Role: FOLLOWER
I20260619 04:34:11.218948 8485 consensus_queue.cc:260] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.219995 8486 tablet_bootstrap.cc:492] T a88c799fec3640ee98baef97250e84d5 P 31b8fdefbb5840d78f3dfb53fb70973a: Bootstrap starting.
I20260619 04:34:11.222079 8486 tablet_bootstrap.cc:654] T a88c799fec3640ee98baef97250e84d5 P 31b8fdefbb5840d78f3dfb53fb70973a: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:11.222205 8485 ts_tablet_manager.cc:1428] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260619 04:34:11.232924 8509 tablet_replica.cc:330] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a: stopping tablet replica
I20260619 04:34:11.233430 8509 raft_consensus.cc:2229] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:34:11.234093 8509 raft_consensus.cc:2258] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:34:11.236091 8486 tablet_bootstrap.cc:492] T a88c799fec3640ee98baef97250e84d5 P 31b8fdefbb5840d78f3dfb53fb70973a: No bootstrap required, opened a new log
I20260619 04:34:11.236318 8486 ts_tablet_manager.cc:1397] T a88c799fec3640ee98baef97250e84d5 P 31b8fdefbb5840d78f3dfb53fb70973a: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.007s
I20260619 04:34:11.236493 8509 ts_tablet_manager.cc:1905] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:11.237238 8486 raft_consensus.cc:348] T a88c799fec3640ee98baef97250e84d5 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.237536 8486 raft_consensus.cc:374] T a88c799fec3640ee98baef97250e84d5 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:11.237682 8486 raft_consensus.cc:729] T a88c799fec3640ee98baef97250e84d5 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b8fdefbb5840d78f3dfb53fb70973a, State: Initialized, Role: FOLLOWER
I20260619 04:34:11.239099 8486 consensus_queue.cc:260] T a88c799fec3640ee98baef97250e84d5 P 31b8fdefbb5840d78f3dfb53fb70973a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.239832 8508 ts_tablet_manager.cc:1905] T 319f290b8a8f40c28f79ee509f5aae20 P eeafa8fcfab041fb8fb588988c4aa694: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:11.242051 8486 ts_tablet_manager.cc:1428] T a88c799fec3640ee98baef97250e84d5 P 31b8fdefbb5840d78f3dfb53fb70973a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260619 04:34:11.242520 8510 ts_tablet_manager.cc:1918] T 319f290b8a8f40c28f79ee509f5aae20 P b72019478929468888fcd697a4d4968a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:34:11.242679 8510 log.cc:1196] T 319f290b8a8f40c28f79ee509f5aae20 P b72019478929468888fcd697a4d4968a: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/319f290b8a8f40c28f79ee509f5aae20
I20260619 04:34:11.243148 8510 ts_tablet_manager.cc:1939] T 319f290b8a8f40c28f79ee509f5aae20 P b72019478929468888fcd697a4d4968a: Deleting consensus metadata
I20260619 04:34:11.245059 7960 catalog_manager.cc:4841] TS b72019478929468888fcd697a4d4968a (127.0.97.3:38111): tablet 319f290b8a8f40c28f79ee509f5aae20 (table test-table [id=a7abc34ae59c4730b92b656049f62fae]) successfully deleted
I20260619 04:34:11.246017 8484 tablet_bootstrap.cc:492] T a88c799fec3640ee98baef97250e84d5 P b72019478929468888fcd697a4d4968a: No bootstrap required, opened a new log
I20260619 04:34:11.246250 8484 ts_tablet_manager.cc:1397] T a88c799fec3640ee98baef97250e84d5 P b72019478929468888fcd697a4d4968a: Time spent bootstrapping tablet: real 0.035s user 0.015s sys 0.000s
I20260619 04:34:11.246382 8508 ts_tablet_manager.cc:1918] T 319f290b8a8f40c28f79ee509f5aae20 P eeafa8fcfab041fb8fb588988c4aa694: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:34:11.246580 8508 log.cc:1196] T 319f290b8a8f40c28f79ee509f5aae20 P eeafa8fcfab041fb8fb588988c4aa694: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-0/wal/wals/319f290b8a8f40c28f79ee509f5aae20
I20260619 04:34:11.247133 8508 ts_tablet_manager.cc:1939] T 319f290b8a8f40c28f79ee509f5aae20 P eeafa8fcfab041fb8fb588988c4aa694: Deleting consensus metadata
I20260619 04:34:11.247076 8484 raft_consensus.cc:348] T a88c799fec3640ee98baef97250e84d5 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.247321 8484 raft_consensus.cc:374] T a88c799fec3640ee98baef97250e84d5 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:11.247439 8484 raft_consensus.cc:729] T a88c799fec3640ee98baef97250e84d5 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b72019478929468888fcd697a4d4968a, State: Initialized, Role: FOLLOWER
I20260619 04:34:11.247728 8484 consensus_queue.cc:260] T a88c799fec3640ee98baef97250e84d5 P b72019478929468888fcd697a4d4968a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.249379 7961 catalog_manager.cc:4841] TS eeafa8fcfab041fb8fb588988c4aa694 (127.0.97.1:35349): tablet 319f290b8a8f40c28f79ee509f5aae20 (table test-table [id=a7abc34ae59c4730b92b656049f62fae]) successfully deleted
I20260619 04:34:11.250356 8484 ts_tablet_manager.cc:1428] T a88c799fec3640ee98baef97250e84d5 P b72019478929468888fcd697a4d4968a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 04:34:11.253494 8509 ts_tablet_manager.cc:1918] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:34:11.253784 8509 log.cc:1196] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/319f290b8a8f40c28f79ee509f5aae20
I20260619 04:34:11.254333 8509 ts_tablet_manager.cc:1939] T 319f290b8a8f40c28f79ee509f5aae20 P 31b8fdefbb5840d78f3dfb53fb70973a: Deleting consensus metadata
I20260619 04:34:11.256119 7962 catalog_manager.cc:4841] TS 31b8fdefbb5840d78f3dfb53fb70973a (127.0.97.2:36779): tablet 319f290b8a8f40c28f79ee509f5aae20 (table test-table [id=a7abc34ae59c4730b92b656049f62fae]) successfully deleted
I20260619 04:34:11.296976 8491 raft_consensus.cc:482] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:34:11.297231 8491 raft_consensus.cc:504] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.299124 8491 leader_election.cc:288] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31b8fdefbb5840d78f3dfb53fb70973a (127.0.97.2:36779), b72019478929468888fcd697a4d4968a (127.0.97.3:38111)
I20260619 04:34:11.307621 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:11.300141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54055 (local address 127.0.97.3:38111)
0619 04:34:11.300312 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:11.300318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:11.301284 (+ 966us) server_negotiation.cc:407] Connection header received
0619 04:34:11.301419 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:11.301425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:11.301510 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:11.301602 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:11.302197 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.302904 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.305327 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.306484 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.306601 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:11.306712 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:11.307233 (+ 521us) server_negotiation.cc:299] Negotiation successful
0619 04:34:11.307362 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 04:34:11.307749 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:11.299565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38111 (local address 127.0.97.1:54055)
0619 04:34:11.301164 (+ 1599us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:11.301181 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:11.301303 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:11.301646 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:11.301650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:11.301669 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:11.302052 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:11.302061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.303028 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.303033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:11.305156 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:11.305171 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.306840 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.306846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:11.307005 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:11.307041 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 04:34:11.307179 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 04:34:11.307329 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1462,"mutex_wait_us":4,"thread_start_us":1280,"threads_started":1}
I20260619 04:34:11.308027 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:11.299822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35063 (local address 127.0.97.2:36779)
0619 04:34:11.300000 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:11.300006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:11.300902 (+ 896us) server_negotiation.cc:407] Connection header received
0619 04:34:11.301212 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:11.301218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:11.301302 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:11.301410 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:11.302097 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.303302 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.305640 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.306865 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.306984 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:11.307072 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:11.307456 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 04:34:11.307573 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 04:34:11.308547 8410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88c799fec3640ee98baef97250e84d5" candidate_uuid: "eeafa8fcfab041fb8fb588988c4aa694" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b72019478929468888fcd697a4d4968a" is_pre_election: true
I20260619 04:34:11.308637 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:11.299428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36779 (local address 127.0.97.1:35063)
0619 04:34:11.300788 (+ 1360us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:11.300811 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:11.301031 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:11.301442 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:11.301446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:11.301470 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:11.301890 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:11.301904 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.303452 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.303457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:11.305462 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:11.305475 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.307115 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.307122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:11.307287 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:11.307318 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 04:34:11.308116 (+ 798us) client_negotiation.cc:240] Negotiation successful
0619 04:34:11.308255 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1225,"mutex_wait_us":22,"thread_start_us":1052,"threads_started":1}
I20260619 04:34:11.308784 8410 raft_consensus.cc:2454] T a88c799fec3640ee98baef97250e84d5 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeafa8fcfab041fb8fb588988c4aa694 in term 0.
I20260619 04:34:11.309199 8063 leader_election.cc:302] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b72019478929468888fcd697a4d4968a, eeafa8fcfab041fb8fb588988c4aa694; no voters:
I20260619 04:34:11.309614 8491 raft_consensus.cc:2781] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:34:11.309733 8491 raft_consensus.cc:482] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:34:11.309548 8271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88c799fec3640ee98baef97250e84d5" candidate_uuid: "eeafa8fcfab041fb8fb588988c4aa694" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" is_pre_election: true
I20260619 04:34:11.309834 8491 raft_consensus.cc:3037] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:11.309976 8271 raft_consensus.cc:2454] T a88c799fec3640ee98baef97250e84d5 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeafa8fcfab041fb8fb588988c4aa694 in term 0.
I20260619 04:34:11.313102 8491 raft_consensus.cc:504] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.313820 8491 leader_election.cc:288] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694 [CANDIDATE]: Term 1 election: Requested vote from peers 31b8fdefbb5840d78f3dfb53fb70973a (127.0.97.2:36779), b72019478929468888fcd697a4d4968a (127.0.97.3:38111)
I20260619 04:34:11.314327 8271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88c799fec3640ee98baef97250e84d5" candidate_uuid: "eeafa8fcfab041fb8fb588988c4aa694" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b8fdefbb5840d78f3dfb53fb70973a"
I20260619 04:34:11.314347 8410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88c799fec3640ee98baef97250e84d5" candidate_uuid: "eeafa8fcfab041fb8fb588988c4aa694" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b72019478929468888fcd697a4d4968a"
I20260619 04:34:11.314504 8271 raft_consensus.cc:3037] T a88c799fec3640ee98baef97250e84d5 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:11.314538 8410 raft_consensus.cc:3037] T a88c799fec3640ee98baef97250e84d5 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:11.316720 8271 raft_consensus.cc:2454] T a88c799fec3640ee98baef97250e84d5 P 31b8fdefbb5840d78f3dfb53fb70973a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeafa8fcfab041fb8fb588988c4aa694 in term 1.
I20260619 04:34:11.317116 8065 leader_election.cc:302] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b8fdefbb5840d78f3dfb53fb70973a, eeafa8fcfab041fb8fb588988c4aa694; no voters:
I20260619 04:34:11.317381 8491 raft_consensus.cc:2781] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:34:11.317375 8410 raft_consensus.cc:2454] T a88c799fec3640ee98baef97250e84d5 P b72019478929468888fcd697a4d4968a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeafa8fcfab041fb8fb588988c4aa694 in term 1.
I20260619 04:34:11.318027 8491 raft_consensus.cc:686] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694 [term 1 LEADER]: Becoming Leader. State: Replica: eeafa8fcfab041fb8fb588988c4aa694, State: Running, Role: LEADER
I20260619 04:34:11.318380 8491 consensus_queue.cc:237] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.321791 7974 catalog_manager.cc:5495] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694 reported cstate change: term changed from 0 to 1, leader changed from <none> to eeafa8fcfab041fb8fb588988c4aa694 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "eeafa8fcfab041fb8fb588988c4aa694" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } health_report { overall_health: UNKNOWN } } }
Jun 19 04:34:11 dist-test-slave-n3tq krb5kdc[7930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843649, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 04:34:11.361063 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:11.346500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38229 (local address 127.0.0.1:58998)
0619 04:34:11.346955 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:11.346983 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:11.347123 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:11.347511 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:11.347516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:11.348308 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:11.348568 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:11.348580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.349643 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.349648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:11.351081 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:11.351095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.351714 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.351721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:11.351890 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:11.352651 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:11.352678 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:11.355297 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:11.358827 (+ 3530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:11.358836 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:11.358842 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:11.359236 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:11.359591 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:11.359594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:11.359596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:11.359725 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:11.360270 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:11.360276 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:11.360421 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 04:34:11.360588 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 04:34:11.360803 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":211,"threads_started":1}
I20260619 04:34:11.361368 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:11.346592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58998 (local address 127.0.97.62:38229)
0619 04:34:11.347244 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:11.347248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:11.347263 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:34:11.347305 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:11.347308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:11.347363 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:11.347454 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:11.348725 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.349501 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.351209 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.351578 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.351677 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:11.355496 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:11.355503 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:11.355508 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:11.355543 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:11.358701 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:11.359348 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:11.359353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:11.359357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:11.359488 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:11.359816 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:11.359821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:11.359825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:11.360160 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:11.360317 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:11.360698 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 04:34:11.360949 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":559,"thread_start_us":453,"threads_started":1}
I20260619 04:34:11.366357 7974 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58998:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 04:34:11.367295 7974 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:34:11.373708 8111 tablet_service.cc:1467] Processing CreateTablet for tablet 74f2e08ec15147d1a076eaa480d56fd5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a993e0ad84094155a2e7739d1a745244]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 04:34:11.373862 8251 tablet_service.cc:1467] Processing CreateTablet for tablet 74f2e08ec15147d1a076eaa480d56fd5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a993e0ad84094155a2e7739d1a745244]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 04:34:11.374061 8389 tablet_service.cc:1467] Processing CreateTablet for tablet 74f2e08ec15147d1a076eaa480d56fd5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a993e0ad84094155a2e7739d1a745244]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 04:34:11.374254 8111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f2e08ec15147d1a076eaa480d56fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:11.374475 8389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f2e08ec15147d1a076eaa480d56fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:11.374468 8251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f2e08ec15147d1a076eaa480d56fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:11.381242 8485 tablet_bootstrap.cc:492] T 74f2e08ec15147d1a076eaa480d56fd5 P eeafa8fcfab041fb8fb588988c4aa694: Bootstrap starting.
I20260619 04:34:11.381253 8486 tablet_bootstrap.cc:492] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a: Bootstrap starting.
I20260619 04:34:11.382190 8484 tablet_bootstrap.cc:492] T 74f2e08ec15147d1a076eaa480d56fd5 P b72019478929468888fcd697a4d4968a: Bootstrap starting.
I20260619 04:34:11.382714 8486 tablet_bootstrap.cc:654] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:11.382709 8485 tablet_bootstrap.cc:654] T 74f2e08ec15147d1a076eaa480d56fd5 P eeafa8fcfab041fb8fb588988c4aa694: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:11.383711 8484 tablet_bootstrap.cc:654] T 74f2e08ec15147d1a076eaa480d56fd5 P b72019478929468888fcd697a4d4968a: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:11.384630 8486 tablet_bootstrap.cc:492] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a: No bootstrap required, opened a new log
I20260619 04:34:11.384799 8486 ts_tablet_manager.cc:1397] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260619 04:34:11.385277 8485 tablet_bootstrap.cc:492] T 74f2e08ec15147d1a076eaa480d56fd5 P eeafa8fcfab041fb8fb588988c4aa694: No bootstrap required, opened a new log
I20260619 04:34:11.385411 8485 ts_tablet_manager.cc:1397] T 74f2e08ec15147d1a076eaa480d56fd5 P eeafa8fcfab041fb8fb588988c4aa694: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 04:34:11.385586 8486 raft_consensus.cc:348] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.385816 8486 raft_consensus.cc:374] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:11.385891 8486 raft_consensus.cc:729] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b8fdefbb5840d78f3dfb53fb70973a, State: Initialized, Role: FOLLOWER
I20260619 04:34:11.386029 8485 raft_consensus.cc:348] T 74f2e08ec15147d1a076eaa480d56fd5 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.386240 8485 raft_consensus.cc:374] T 74f2e08ec15147d1a076eaa480d56fd5 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:11.386134 8486 consensus_queue.cc:260] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.386303 8485 raft_consensus.cc:729] T 74f2e08ec15147d1a076eaa480d56fd5 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeafa8fcfab041fb8fb588988c4aa694, State: Initialized, Role: FOLLOWER
I20260619 04:34:11.386693 8486 ts_tablet_manager.cc:1428] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 04:34:11.386566 8485 consensus_queue.cc:260] T 74f2e08ec15147d1a076eaa480d56fd5 P eeafa8fcfab041fb8fb588988c4aa694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.386947 8491 tablet_replica.cc:439] T 74f2e08ec15147d1a076eaa480d56fd5 P eeafa8fcfab041fb8fb588988c4aa694: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } } }
I20260619 04:34:11.387189 8491 tablet_replica.cc:442] T 74f2e08ec15147d1a076eaa480d56fd5 P eeafa8fcfab041fb8fb588988c4aa694: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:34:11.387037 8503 tablet_replica.cc:439] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } } }
I20260619 04:34:11.387363 8503 tablet_replica.cc:442] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:34:11.390116 8485 ts_tablet_manager.cc:1428] T 74f2e08ec15147d1a076eaa480d56fd5 P eeafa8fcfab041fb8fb588988c4aa694: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260619 04:34:11.394234 8484 tablet_bootstrap.cc:492] T 74f2e08ec15147d1a076eaa480d56fd5 P b72019478929468888fcd697a4d4968a: No bootstrap required, opened a new log
I20260619 04:34:11.394408 8484 ts_tablet_manager.cc:1397] T 74f2e08ec15147d1a076eaa480d56fd5 P b72019478929468888fcd697a4d4968a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20260619 04:34:11.395094 8484 raft_consensus.cc:348] T 74f2e08ec15147d1a076eaa480d56fd5 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.395337 8484 raft_consensus.cc:374] T 74f2e08ec15147d1a076eaa480d56fd5 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:11.395411 8484 raft_consensus.cc:729] T 74f2e08ec15147d1a076eaa480d56fd5 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b72019478929468888fcd697a4d4968a, State: Initialized, Role: FOLLOWER
I20260619 04:34:11.395732 8484 consensus_queue.cc:260] T 74f2e08ec15147d1a076eaa480d56fd5 P b72019478929468888fcd697a4d4968a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.396272 8484 ts_tablet_manager.cc:1428] T 74f2e08ec15147d1a076eaa480d56fd5 P b72019478929468888fcd697a4d4968a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 04:34:11.396185 8489 tablet_replica.cc:439] T 74f2e08ec15147d1a076eaa480d56fd5 P b72019478929468888fcd697a4d4968a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } } }
I20260619 04:34:11.396565 8489 tablet_replica.cc:442] T 74f2e08ec15147d1a076eaa480d56fd5 P b72019478929468888fcd697a4d4968a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:34:11.555799 8503 raft_consensus.cc:482] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:34:11.556059 8503 raft_consensus.cc:504] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.556953 8503 leader_election.cc:288] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eeafa8fcfab041fb8fb588988c4aa694 (127.0.97.1:35349), b72019478929468888fcd697a4d4968a (127.0.97.3:38111)
I20260619 04:34:11.557444 8410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f2e08ec15147d1a076eaa480d56fd5" candidate_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b72019478929468888fcd697a4d4968a" is_pre_election: true
I20260619 04:34:11.557487 8132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f2e08ec15147d1a076eaa480d56fd5" candidate_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeafa8fcfab041fb8fb588988c4aa694" is_pre_election: true
I20260619 04:34:11.557684 8410 raft_consensus.cc:2454] T 74f2e08ec15147d1a076eaa480d56fd5 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b8fdefbb5840d78f3dfb53fb70973a in term 0.
I20260619 04:34:11.557745 8132 raft_consensus.cc:2454] T 74f2e08ec15147d1a076eaa480d56fd5 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b8fdefbb5840d78f3dfb53fb70973a in term 0.
I20260619 04:34:11.558141 8203 leader_election.cc:302] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b8fdefbb5840d78f3dfb53fb70973a, b72019478929468888fcd697a4d4968a; no voters:
I20260619 04:34:11.558449 8503 raft_consensus.cc:2781] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:34:11.558574 8503 raft_consensus.cc:482] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:34:11.558663 8503 raft_consensus.cc:3037] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:11.560994 8503 raft_consensus.cc:504] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.561494 8503 leader_election.cc:288] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a [CANDIDATE]: Term 1 election: Requested vote from peers eeafa8fcfab041fb8fb588988c4aa694 (127.0.97.1:35349), b72019478929468888fcd697a4d4968a (127.0.97.3:38111)
I20260619 04:34:11.562006 8132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f2e08ec15147d1a076eaa480d56fd5" candidate_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeafa8fcfab041fb8fb588988c4aa694"
I20260619 04:34:11.561993 8410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f2e08ec15147d1a076eaa480d56fd5" candidate_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b72019478929468888fcd697a4d4968a"
I20260619 04:34:11.562206 8132 raft_consensus.cc:3037] T 74f2e08ec15147d1a076eaa480d56fd5 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:11.562217 8410 raft_consensus.cc:3037] T 74f2e08ec15147d1a076eaa480d56fd5 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:11.564991 8410 raft_consensus.cc:2454] T 74f2e08ec15147d1a076eaa480d56fd5 P b72019478929468888fcd697a4d4968a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b8fdefbb5840d78f3dfb53fb70973a in term 1.
I20260619 04:34:11.565007 8132 raft_consensus.cc:2454] T 74f2e08ec15147d1a076eaa480d56fd5 P eeafa8fcfab041fb8fb588988c4aa694 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b8fdefbb5840d78f3dfb53fb70973a in term 1.
I20260619 04:34:11.565399 8204 leader_election.cc:302] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b8fdefbb5840d78f3dfb53fb70973a, eeafa8fcfab041fb8fb588988c4aa694; no voters:
I20260619 04:34:11.565694 8503 raft_consensus.cc:2781] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:34:11.565850 8503 raft_consensus.cc:686] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a [term 1 LEADER]: Becoming Leader. State: Replica: 31b8fdefbb5840d78f3dfb53fb70973a, State: Running, Role: LEADER
I20260619 04:34:11.566170 8503 consensus_queue.cc:237] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.567866 8495 tablet_replica.cc:439] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a: TxnStatusTablet state changed. Reason: New leader 31b8fdefbb5840d78f3dfb53fb70973a. Latest consensus state: current_term: 1 leader_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } } }
I20260619 04:34:11.568261 8495 tablet_replica.cc:442] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a: This TxnStatusTablet replica's current role is: LEADER
I20260619 04:34:11.569598 7975 catalog_manager.cc:5495] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a reported cstate change: term changed from 0 to 1, leader changed from <none> to 31b8fdefbb5840d78f3dfb53fb70973a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } health_report { overall_health: UNKNOWN } } }
I20260619 04:34:11.578233 8529 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 04:34:11.578459 8529 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 04:34:11.589005 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:11.583726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36779 (local address 127.0.0.1:59334)
0619 04:34:11.584020 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:11.584035 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:11.584142 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:11.584551 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:11.584554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:11.584564 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:34:11.584750 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:11.584762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.586164 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.586168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:11.587133 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:11.587143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.588056 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.588061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:11.588214 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:11.588266 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:34:11.588612 (+ 346us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:34:11.588623 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 04:34:11.588684 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 04:34:11.588788 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260619 04:34:11.589036 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:11.583733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59334 (local address 127.0.97.2:36779)
0619 04:34:11.583919 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:11.583926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:11.584095 (+ 169us) server_negotiation.cc:407] Connection header received
0619 04:34:11.584310 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:11.584315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:11.584405 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:11.584537 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:34:11.584876 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.586038 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.587277 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.587930 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.588043 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:11.588403 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:34:11.588537 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:34:11.588613 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:11.588712 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 04:34:11.588783 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 04:34:11.592209 8132 raft_consensus.cc:1261] T 74f2e08ec15147d1a076eaa480d56fd5 P eeafa8fcfab041fb8fb588988c4aa694 [term 1 FOLLOWER]: Refusing update from remote peer 31b8fdefbb5840d78f3dfb53fb70973a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:34:11.592243 8410 raft_consensus.cc:1261] T 74f2e08ec15147d1a076eaa480d56fd5 P b72019478929468888fcd697a4d4968a [term 1 FOLLOWER]: Refusing update from remote peer 31b8fdefbb5840d78f3dfb53fb70973a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:34:11.592777 8503 consensus_queue.cc:1035] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:34:11.593060 8495 consensus_queue.cc:1035] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:34:11.596015 8491 tablet_replica.cc:439] T 74f2e08ec15147d1a076eaa480d56fd5 P eeafa8fcfab041fb8fb588988c4aa694: TxnStatusTablet state changed. Reason: New leader 31b8fdefbb5840d78f3dfb53fb70973a. Latest consensus state: current_term: 1 leader_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } } }
I20260619 04:34:11.596310 8491 tablet_replica.cc:442] T 74f2e08ec15147d1a076eaa480d56fd5 P eeafa8fcfab041fb8fb588988c4aa694: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:34:11.597649 8495 tablet_replica.cc:439] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } } }
I20260619 04:34:11.597950 8498 mvcc.cc:205] Tried to move back new op lower bound from 7298431596916375552 to 7298431596817506304. Current Snapshot: MvccSnapshot[applied={T|T < 7298431596916375552 or (T in {7298431596916375552})}]
I20260619 04:34:11.599026 8495 tablet_replica.cc:442] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a: This TxnStatusTablet replica's current role is: LEADER
I20260619 04:34:11.599157 8500 mvcc.cc:205] Tried to move back new op lower bound from 7298431596916375552 to 7298431596817506304. Current Snapshot: MvccSnapshot[applied={T|T < 7298431596916375552}]
I20260619 04:34:11.599401 8491 tablet_replica.cc:439] T 74f2e08ec15147d1a076eaa480d56fd5 P eeafa8fcfab041fb8fb588988c4aa694: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } } }
I20260619 04:34:11.599623 8491 tablet_replica.cc:442] T 74f2e08ec15147d1a076eaa480d56fd5 P eeafa8fcfab041fb8fb588988c4aa694: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:34:11.600273 8489 tablet_replica.cc:439] T 74f2e08ec15147d1a076eaa480d56fd5 P b72019478929468888fcd697a4d4968a: TxnStatusTablet state changed. Reason: New leader 31b8fdefbb5840d78f3dfb53fb70973a. Latest consensus state: current_term: 1 leader_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } } }
I20260619 04:34:11.600800 8489 tablet_replica.cc:442] T 74f2e08ec15147d1a076eaa480d56fd5 P b72019478929468888fcd697a4d4968a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:34:11.605218 8492 tablet_replica.cc:439] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } } }
I20260619 04:34:11.605487 8492 tablet_replica.cc:442] T 74f2e08ec15147d1a076eaa480d56fd5 P 31b8fdefbb5840d78f3dfb53fb70973a: This TxnStatusTablet replica's current role is: LEADER
I20260619 04:34:11.607285 8489 tablet_replica.cc:439] T 74f2e08ec15147d1a076eaa480d56fd5 P b72019478929468888fcd697a4d4968a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } } }
I20260619 04:34:11.607515 8489 tablet_replica.cc:442] T 74f2e08ec15147d1a076eaa480d56fd5 P b72019478929468888fcd697a4d4968a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:34:11.617786 8533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:11.607204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.0.97.1:35349)
0619 04:34:11.612911 (+ 5707us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:11.612916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:11.612936 (+ 20us) server_negotiation.cc:407] Connection header received
0619 04:34:11.613016 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:11.613020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:11.613093 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:11.613263 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:34:11.613664 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.614427 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.615669 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.616071 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.616181 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:11.616671 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:34:11.617192 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:34:11.617323 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:11.617415 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 04:34:11.617490 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5588,"thread_start_us":5477,"threads_started":1}
I20260619 04:34:11.634232 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:11.627901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47219 (local address 127.0.97.2:36779)
0619 04:34:11.628111 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:11.628118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:11.628639 (+ 521us) server_negotiation.cc:407] Connection header received
0619 04:34:11.628825 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:11.628830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:11.628921 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:11.629017 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:34:11.629736 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.630629 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.632565 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.633012 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.633106 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:11.633555 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:34:11.633695 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:34:11.633763 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:11.633930 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 04:34:11.633998 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 04:34:11.634344 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:11.627873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36779 (local address 127.0.97.1:47219)
0619 04:34:11.628524 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:11.628563 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:11.628702 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:11.629169 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:11.629174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:11.629187 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:34:11.629549 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:11.629560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.630764 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.630769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:11.632395 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:11.632411 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.633178 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.633184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:11.633335 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:11.633400 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:34:11.633782 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:34:11.633792 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 04:34:11.633865 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 04:34:11.633988 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":342,"threads_started":1}
I20260619 04:34:11.643739 8410 raft_consensus.cc:1261] T a88c799fec3640ee98baef97250e84d5 P b72019478929468888fcd697a4d4968a [term 1 FOLLOWER]: Refusing update from remote peer eeafa8fcfab041fb8fb588988c4aa694: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:34:11.643927 8271 raft_consensus.cc:1261] T a88c799fec3640ee98baef97250e84d5 P 31b8fdefbb5840d78f3dfb53fb70973a [term 1 FOLLOWER]: Refusing update from remote peer eeafa8fcfab041fb8fb588988c4aa694: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:34:11.644415 8516 consensus_queue.cc:1035] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:34:11.645082 8491 consensus_queue.cc:1035] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:34:11.677421 8541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:11.670516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35349 (local address 127.0.97.2:59657)
0619 04:34:11.671202 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:11.671222 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:11.671370 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:11.671693 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:11.671698 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:11.671721 (+ 23us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:34:11.672115 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:11.672128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.673545 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.673551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:11.675336 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:11.675348 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.676211 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.676216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:11.676334 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:11.676387 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:34:11.676829 (+ 442us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:34:11.676838 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 04:34:11.676912 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 04:34:11.677044 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":503,"threads_started":1}
I20260619 04:34:11.677992 8533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:11.670837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59657 (local address 127.0.97.1:35349)
0619 04:34:11.671008 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:11.671015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:11.671317 (+ 302us) server_negotiation.cc:407] Connection header received
0619 04:34:11.671491 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:11.671497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:11.671578 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:11.671669 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:34:11.672271 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.673431 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.675453 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.676055 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.676178 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:11.676598 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:34:11.676734 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:34:11.676823 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:11.676931 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 04:34:11.677198 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 04:34:11.714140 7975 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:34:11.714388 7975 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:34:11.717648 7975 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74: Sending DeleteTablet for 3 replicas of tablet a88c799fec3640ee98baef97250e84d5
I20260619 04:34:11.718362 8111 tablet_service.cc:1514] Processing DeleteTablet for tablet a88c799fec3640ee98baef97250e84d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:34:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55310
I20260619 04:34:11.718611 8508 tablet_replica.cc:330] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694: stopping tablet replica
I20260619 04:34:11.718806 8508 raft_consensus.cc:2229] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:34:11.718806 8389 tablet_service.cc:1514] Processing DeleteTablet for tablet a88c799fec3640ee98baef97250e84d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:34:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40630
I20260619 04:34:11.719046 8510 tablet_replica.cc:330] T a88c799fec3640ee98baef97250e84d5 P b72019478929468888fcd697a4d4968a: stopping tablet replica
I20260619 04:34:11.719223 8510 raft_consensus.cc:2229] T a88c799fec3640ee98baef97250e84d5 P b72019478929468888fcd697a4d4968a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:34:11.719260 8508 raft_consensus.cc:2258] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:34:11.719398 8510 raft_consensus.cc:2258] T a88c799fec3640ee98baef97250e84d5 P b72019478929468888fcd697a4d4968a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:34:11.719411 8251 tablet_service.cc:1514] Processing DeleteTablet for tablet a88c799fec3640ee98baef97250e84d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:34:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59324
I20260619 04:34:11.720156 8509 tablet_replica.cc:330] T a88c799fec3640ee98baef97250e84d5 P 31b8fdefbb5840d78f3dfb53fb70973a: stopping tablet replica
I20260619 04:34:11.720360 8509 raft_consensus.cc:2229] T a88c799fec3640ee98baef97250e84d5 P 31b8fdefbb5840d78f3dfb53fb70973a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:34:11.720546 8508 ts_tablet_manager.cc:1905] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:11.720556 8509 raft_consensus.cc:2258] T a88c799fec3640ee98baef97250e84d5 P 31b8fdefbb5840d78f3dfb53fb70973a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:34:11.722043 8510 ts_tablet_manager.cc:1905] T a88c799fec3640ee98baef97250e84d5 P b72019478929468888fcd697a4d4968a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:11.725679 8509 ts_tablet_manager.cc:1905] T a88c799fec3640ee98baef97250e84d5 P 31b8fdefbb5840d78f3dfb53fb70973a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:11.727562 8508 ts_tablet_manager.cc:1918] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:34:11.727696 8508 log.cc:1196] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-0/wal/wals/a88c799fec3640ee98baef97250e84d5
I20260619 04:34:11.728119 8508 ts_tablet_manager.cc:1939] T a88c799fec3640ee98baef97250e84d5 P eeafa8fcfab041fb8fb588988c4aa694: Deleting consensus metadata
I20260619 04:34:11.729187 7961 catalog_manager.cc:4841] TS eeafa8fcfab041fb8fb588988c4aa694 (127.0.97.1:35349): tablet a88c799fec3640ee98baef97250e84d5 (table test-table [id=28ee7b18577d4a64a918f38fef1021d9]) successfully deleted
I20260619 04:34:11.745194 8510 ts_tablet_manager.cc:1918] T a88c799fec3640ee98baef97250e84d5 P b72019478929468888fcd697a4d4968a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:34:11.745407 8509 ts_tablet_manager.cc:1918] T a88c799fec3640ee98baef97250e84d5 P 31b8fdefbb5840d78f3dfb53fb70973a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:34:11.745577 8509 log.cc:1196] T a88c799fec3640ee98baef97250e84d5 P 31b8fdefbb5840d78f3dfb53fb70973a: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/a88c799fec3640ee98baef97250e84d5
I20260619 04:34:11.745579 8510 log.cc:1196] T a88c799fec3640ee98baef97250e84d5 P b72019478929468888fcd697a4d4968a: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/a88c799fec3640ee98baef97250e84d5
I20260619 04:34:11.745950 8509 ts_tablet_manager.cc:1939] T a88c799fec3640ee98baef97250e84d5 P 31b8fdefbb5840d78f3dfb53fb70973a: Deleting consensus metadata
I20260619 04:34:11.746134 8510 ts_tablet_manager.cc:1939] T a88c799fec3640ee98baef97250e84d5 P b72019478929468888fcd697a4d4968a: Deleting consensus metadata
I20260619 04:34:11.747044 7962 catalog_manager.cc:4841] TS 31b8fdefbb5840d78f3dfb53fb70973a (127.0.97.2:36779): tablet a88c799fec3640ee98baef97250e84d5 (table test-table [id=28ee7b18577d4a64a918f38fef1021d9]) successfully deleted
I20260619 04:34:11.747184 7960 catalog_manager.cc:4841] TS b72019478929468888fcd697a4d4968a (127.0.97.3:38111): tablet a88c799fec3640ee98baef97250e84d5 (table test-table [id=28ee7b18577d4a64a918f38fef1021d9]) successfully deleted
Jun 19 04:34:11 dist-test-slave-n3tq krb5kdc[7930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 04:34:11 dist-test-slave-n3tq krb5kdc[7930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:11 dist-test-slave-n3tq krb5kdc[7930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 04:34:11.808270 8524 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:11.798755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59006 (local address 127.0.97.62:38229)
0619 04:34:11.798951 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:11.798957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:11.799480 (+ 523us) server_negotiation.cc:407] Connection header received
0619 04:34:11.799659 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:11.799663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:11.799741 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:11.799826 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:11.801161 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.801862 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.803050 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.803462 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.803571 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:11.807989 (+ 4418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59006: BlockingRecv error: recv got EOF from 127.0.0.1:59006 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 04:34:11 dist-test-slave-n3tq krb5kdc[7930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 04:34:11.828420 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:11.814675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59014 (local address 127.0.97.62:38229)
0619 04:34:11.814872 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:11.814878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:11.815116 (+ 238us) server_negotiation.cc:407] Connection header received
0619 04:34:11.815274 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:11.815279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:11.815344 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:11.815424 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:11.816605 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.817546 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.818800 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.819187 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.819302 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:11.822933 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:11.822937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:11.822941 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:11.822978 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:11.825484 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:11.826137 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:11.826143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:11.826147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:11.826287 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:11.826662 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:11.826666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:11.826669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:11.826999 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:11.827181 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:11.827718 (+ 537us) server_negotiation.cc:299] Negotiation successful
0619 04:34:11.827953 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 04:34:11.832091 7975 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:34:11.832757 7975 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:34:11.839028 8111 tablet_service.cc:1467] Processing CreateTablet for tablet c94866b6bff342348cd104c6aa75246a (DEFAULT_TABLE table=test-table [id=0a50033bfc2641e19783490a73fb0f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:34:11.839485 8111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94866b6bff342348cd104c6aa75246a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:11.839596 8389 tablet_service.cc:1467] Processing CreateTablet for tablet c94866b6bff342348cd104c6aa75246a (DEFAULT_TABLE table=test-table [id=0a50033bfc2641e19783490a73fb0f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:34:11.839629 8251 tablet_service.cc:1467] Processing CreateTablet for tablet c94866b6bff342348cd104c6aa75246a (DEFAULT_TABLE table=test-table [id=0a50033bfc2641e19783490a73fb0f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:34:11.840016 8251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94866b6bff342348cd104c6aa75246a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:11.840029 8389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94866b6bff342348cd104c6aa75246a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:11.845860 8485 tablet_bootstrap.cc:492] T c94866b6bff342348cd104c6aa75246a P eeafa8fcfab041fb8fb588988c4aa694: Bootstrap starting.
I20260619 04:34:11.846076 8484 tablet_bootstrap.cc:492] T c94866b6bff342348cd104c6aa75246a P b72019478929468888fcd697a4d4968a: Bootstrap starting.
I20260619 04:34:11.847146 8486 tablet_bootstrap.cc:492] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a: Bootstrap starting.
I20260619 04:34:11.847358 8485 tablet_bootstrap.cc:654] T c94866b6bff342348cd104c6aa75246a P eeafa8fcfab041fb8fb588988c4aa694: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:11.847550 8484 tablet_bootstrap.cc:654] T c94866b6bff342348cd104c6aa75246a P b72019478929468888fcd697a4d4968a: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:11.848740 8486 tablet_bootstrap.cc:654] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:11.859699 8484 tablet_bootstrap.cc:492] T c94866b6bff342348cd104c6aa75246a P b72019478929468888fcd697a4d4968a: No bootstrap required, opened a new log
I20260619 04:34:11.859916 8484 ts_tablet_manager.cc:1397] T c94866b6bff342348cd104c6aa75246a P b72019478929468888fcd697a4d4968a: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.009s
I20260619 04:34:11.860122 8485 tablet_bootstrap.cc:492] T c94866b6bff342348cd104c6aa75246a P eeafa8fcfab041fb8fb588988c4aa694: No bootstrap required, opened a new log
I20260619 04:34:11.860275 8485 ts_tablet_manager.cc:1397] T c94866b6bff342348cd104c6aa75246a P eeafa8fcfab041fb8fb588988c4aa694: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.006s
I20260619 04:34:11.860841 8484 raft_consensus.cc:348] T c94866b6bff342348cd104c6aa75246a P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.861136 8484 raft_consensus.cc:374] T c94866b6bff342348cd104c6aa75246a P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:11.861236 8484 raft_consensus.cc:729] T c94866b6bff342348cd104c6aa75246a P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b72019478929468888fcd697a4d4968a, State: Initialized, Role: FOLLOWER
I20260619 04:34:11.861207 8485 raft_consensus.cc:348] T c94866b6bff342348cd104c6aa75246a P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.861471 8485 raft_consensus.cc:374] T c94866b6bff342348cd104c6aa75246a P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:11.861557 8485 raft_consensus.cc:729] T c94866b6bff342348cd104c6aa75246a P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeafa8fcfab041fb8fb588988c4aa694, State: Initialized, Role: FOLLOWER
I20260619 04:34:11.862001 8485 consensus_queue.cc:260] T c94866b6bff342348cd104c6aa75246a P eeafa8fcfab041fb8fb588988c4aa694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.862720 8484 consensus_queue.cc:260] T c94866b6bff342348cd104c6aa75246a P b72019478929468888fcd697a4d4968a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.863471 8484 ts_tablet_manager.cc:1428] T c94866b6bff342348cd104c6aa75246a P b72019478929468888fcd697a4d4968a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 04:34:11.865060 8485 ts_tablet_manager.cc:1428] T c94866b6bff342348cd104c6aa75246a P eeafa8fcfab041fb8fb588988c4aa694: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 04:34:11.867388 8486 tablet_bootstrap.cc:492] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a: No bootstrap required, opened a new log
I20260619 04:34:11.867611 8486 ts_tablet_manager.cc:1397] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.007s
I20260619 04:34:11.868440 8486 raft_consensus.cc:348] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.868796 8486 raft_consensus.cc:374] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:11.868932 8486 raft_consensus.cc:729] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b8fdefbb5840d78f3dfb53fb70973a, State: Initialized, Role: FOLLOWER
I20260619 04:34:11.869249 8486 consensus_queue.cc:260] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.870378 8486 ts_tablet_manager.cc:1428] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 04:34:11.919575 8492 raft_consensus.cc:482] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:34:11.919790 8492 raft_consensus.cc:504] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.920607 8492 leader_election.cc:288] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eeafa8fcfab041fb8fb588988c4aa694 (127.0.97.1:35349), b72019478929468888fcd697a4d4968a (127.0.97.3:38111)
I20260619 04:34:11.921186 8132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94866b6bff342348cd104c6aa75246a" candidate_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeafa8fcfab041fb8fb588988c4aa694" is_pre_election: true
I20260619 04:34:11.921293 8410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94866b6bff342348cd104c6aa75246a" candidate_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b72019478929468888fcd697a4d4968a" is_pre_election: true
I20260619 04:34:11.921440 8132 raft_consensus.cc:2454] T c94866b6bff342348cd104c6aa75246a P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b8fdefbb5840d78f3dfb53fb70973a in term 0.
I20260619 04:34:11.921525 8410 raft_consensus.cc:2454] T c94866b6bff342348cd104c6aa75246a P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b8fdefbb5840d78f3dfb53fb70973a in term 0.
I20260619 04:34:11.921916 8204 leader_election.cc:302] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b8fdefbb5840d78f3dfb53fb70973a, eeafa8fcfab041fb8fb588988c4aa694; no voters:
I20260619 04:34:11.922216 8492 raft_consensus.cc:2781] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:34:11.922338 8492 raft_consensus.cc:482] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:34:11.922439 8492 raft_consensus.cc:3037] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:11.925545 8492 raft_consensus.cc:504] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.926788 8132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94866b6bff342348cd104c6aa75246a" candidate_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeafa8fcfab041fb8fb588988c4aa694"
I20260619 04:34:11.926793 8410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94866b6bff342348cd104c6aa75246a" candidate_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b72019478929468888fcd697a4d4968a"
I20260619 04:34:11.926993 8132 raft_consensus.cc:3037] T c94866b6bff342348cd104c6aa75246a P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:11.926995 8410 raft_consensus.cc:3037] T c94866b6bff342348cd104c6aa75246a P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:11.929800 8410 raft_consensus.cc:2454] T c94866b6bff342348cd104c6aa75246a P b72019478929468888fcd697a4d4968a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b8fdefbb5840d78f3dfb53fb70973a in term 1.
I20260619 04:34:11.930255 8132 raft_consensus.cc:2454] T c94866b6bff342348cd104c6aa75246a P eeafa8fcfab041fb8fb588988c4aa694 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b8fdefbb5840d78f3dfb53fb70973a in term 1.
I20260619 04:34:11.931018 8203 leader_election.cc:302] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b8fdefbb5840d78f3dfb53fb70973a, b72019478929468888fcd697a4d4968a; no voters:
I20260619 04:34:11.931340 8495 raft_consensus.cc:2781] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:34:11.931473 8495 raft_consensus.cc:686] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a [term 1 LEADER]: Becoming Leader. State: Replica: 31b8fdefbb5840d78f3dfb53fb70973a, State: Running, Role: LEADER
I20260619 04:34:11.931774 8495 consensus_queue.cc:237] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:11.934039 8492 leader_election.cc:288] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a [CANDIDATE]: Term 1 election: Requested vote from peers eeafa8fcfab041fb8fb588988c4aa694 (127.0.97.1:35349), b72019478929468888fcd697a4d4968a (127.0.97.3:38111)
I20260619 04:34:11.934651 7975 catalog_manager.cc:5495] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a reported cstate change: term changed from 0 to 1, leader changed from <none> to 31b8fdefbb5840d78f3dfb53fb70973a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } health_report { overall_health: UNKNOWN } } }
I20260619 04:34:11.964143 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:11.957908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59344 (local address 127.0.97.2:36779)
0619 04:34:11.958077 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:11.958082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:11.958102 (+ 20us) server_negotiation.cc:407] Connection header received
0619 04:34:11.958180 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:11.958186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:11.958269 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:11.958367 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:34:11.958792 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.959914 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.961936 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:11.962646 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:11.962786 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:11.963241 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:34:11.963383 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:34:11.963473 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:11.963683 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 04:34:11.963811 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:34:11.967347 8410 raft_consensus.cc:1261] T c94866b6bff342348cd104c6aa75246a P b72019478929468888fcd697a4d4968a [term 1 FOLLOWER]: Refusing update from remote peer 31b8fdefbb5840d78f3dfb53fb70973a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:34:11.967558 8132 raft_consensus.cc:1261] T c94866b6bff342348cd104c6aa75246a P eeafa8fcfab041fb8fb588988c4aa694 [term 1 FOLLOWER]: Refusing update from remote peer 31b8fdefbb5840d78f3dfb53fb70973a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:34:11.967942 8495 consensus_queue.cc:1035] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:34:11.968197 8503 consensus_queue.cc:1035] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:34:11.988888 7974 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:34:11.989168 7974 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:34:11.992688 7974 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74: Sending DeleteTablet for 3 replicas of tablet c94866b6bff342348cd104c6aa75246a
I20260619 04:34:11.994058 8111 tablet_service.cc:1514] Processing DeleteTablet for tablet c94866b6bff342348cd104c6aa75246a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:34:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55310
I20260619 04:34:11.994393 8389 tablet_service.cc:1514] Processing DeleteTablet for tablet c94866b6bff342348cd104c6aa75246a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:34:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40630
I20260619 04:34:11.994727 8508 tablet_replica.cc:330] T c94866b6bff342348cd104c6aa75246a P eeafa8fcfab041fb8fb588988c4aa694: stopping tablet replica
I20260619 04:34:11.994941 8508 raft_consensus.cc:2229] T c94866b6bff342348cd104c6aa75246a P eeafa8fcfab041fb8fb588988c4aa694 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:34:11.995182 8508 raft_consensus.cc:2258] T c94866b6bff342348cd104c6aa75246a P eeafa8fcfab041fb8fb588988c4aa694 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:34:11.995240 8510 tablet_replica.cc:330] T c94866b6bff342348cd104c6aa75246a P b72019478929468888fcd697a4d4968a: stopping tablet replica
I20260619 04:34:11.995412 8510 raft_consensus.cc:2229] T c94866b6bff342348cd104c6aa75246a P b72019478929468888fcd697a4d4968a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:34:11.995626 8510 raft_consensus.cc:2258] T c94866b6bff342348cd104c6aa75246a P b72019478929468888fcd697a4d4968a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:34:11.996168 8508 ts_tablet_manager.cc:1905] T c94866b6bff342348cd104c6aa75246a P eeafa8fcfab041fb8fb588988c4aa694: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:11.996636 8251 tablet_service.cc:1514] Processing DeleteTablet for tablet c94866b6bff342348cd104c6aa75246a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:34:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59324
I20260619 04:34:11.996915 8509 tablet_replica.cc:330] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a: stopping tablet replica
I20260619 04:34:11.997087 8509 raft_consensus.cc:2229] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:34:11.997433 8509 raft_consensus.cc:2258] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:34:12.000346 8510 ts_tablet_manager.cc:1905] T c94866b6bff342348cd104c6aa75246a P b72019478929468888fcd697a4d4968a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:12.002455 8509 ts_tablet_manager.cc:1905] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:12.004850 7974 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:34:12.005626 7974 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:34:12.010660 8510 ts_tablet_manager.cc:1918] T c94866b6bff342348cd104c6aa75246a P b72019478929468888fcd697a4d4968a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:34:12.010876 8510 log.cc:1196] T c94866b6bff342348cd104c6aa75246a P b72019478929468888fcd697a4d4968a: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/c94866b6bff342348cd104c6aa75246a
I20260619 04:34:12.011365 8510 ts_tablet_manager.cc:1939] T c94866b6bff342348cd104c6aa75246a P b72019478929468888fcd697a4d4968a: Deleting consensus metadata
I20260619 04:34:12.012612 7960 catalog_manager.cc:4841] TS b72019478929468888fcd697a4d4968a (127.0.97.3:38111): tablet c94866b6bff342348cd104c6aa75246a (table test-table [id=0a50033bfc2641e19783490a73fb0f09]) successfully deleted
I20260619 04:34:12.015105 8111 tablet_service.cc:1467] Processing CreateTablet for tablet acdc482d294d4e8bbb39537a6c6039d3 (DEFAULT_TABLE table=test-table [id=00d4ee4bba0d4e52b284b1f0214af747]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:34:12.015594 8111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acdc482d294d4e8bbb39537a6c6039d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:12.017335 8509 ts_tablet_manager.cc:1918] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:34:12.018157 8508 ts_tablet_manager.cc:1918] T c94866b6bff342348cd104c6aa75246a P eeafa8fcfab041fb8fb588988c4aa694: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:34:12.018584 8509 log.cc:1196] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/c94866b6bff342348cd104c6aa75246a
I20260619 04:34:12.018782 8508 log.cc:1196] T c94866b6bff342348cd104c6aa75246a P eeafa8fcfab041fb8fb588988c4aa694: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-0/wal/wals/c94866b6bff342348cd104c6aa75246a
I20260619 04:34:12.019275 8509 ts_tablet_manager.cc:1939] T c94866b6bff342348cd104c6aa75246a P 31b8fdefbb5840d78f3dfb53fb70973a: Deleting consensus metadata
I20260619 04:34:12.019761 8508 ts_tablet_manager.cc:1939] T c94866b6bff342348cd104c6aa75246a P eeafa8fcfab041fb8fb588988c4aa694: Deleting consensus metadata
I20260619 04:34:12.021216 8389 tablet_service.cc:1467] Processing CreateTablet for tablet acdc482d294d4e8bbb39537a6c6039d3 (DEFAULT_TABLE table=test-table [id=00d4ee4bba0d4e52b284b1f0214af747]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:34:12.021679 8389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acdc482d294d4e8bbb39537a6c6039d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:12.022266 8251 tablet_service.cc:1467] Processing CreateTablet for tablet acdc482d294d4e8bbb39537a6c6039d3 (DEFAULT_TABLE table=test-table [id=00d4ee4bba0d4e52b284b1f0214af747]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:34:12.022720 8251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acdc482d294d4e8bbb39537a6c6039d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:12.025332 8485 tablet_bootstrap.cc:492] T acdc482d294d4e8bbb39537a6c6039d3 P eeafa8fcfab041fb8fb588988c4aa694: Bootstrap starting.
I20260619 04:34:12.027179 7962 catalog_manager.cc:4841] TS 31b8fdefbb5840d78f3dfb53fb70973a (127.0.97.2:36779): tablet c94866b6bff342348cd104c6aa75246a (table test-table [id=0a50033bfc2641e19783490a73fb0f09]) successfully deleted
I20260619 04:34:12.027403 8485 tablet_bootstrap.cc:654] T acdc482d294d4e8bbb39537a6c6039d3 P eeafa8fcfab041fb8fb588988c4aa694: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:12.030148 8484 tablet_bootstrap.cc:492] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a: Bootstrap starting.
I20260619 04:34:12.031796 8484 tablet_bootstrap.cc:654] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:12.032068 8486 tablet_bootstrap.cc:492] T acdc482d294d4e8bbb39537a6c6039d3 P 31b8fdefbb5840d78f3dfb53fb70973a: Bootstrap starting.
I20260619 04:34:12.033032 7961 catalog_manager.cc:4841] TS eeafa8fcfab041fb8fb588988c4aa694 (127.0.97.1:35349): tablet c94866b6bff342348cd104c6aa75246a (table test-table [id=0a50033bfc2641e19783490a73fb0f09]) successfully deleted
I20260619 04:34:12.033967 8486 tablet_bootstrap.cc:654] T acdc482d294d4e8bbb39537a6c6039d3 P 31b8fdefbb5840d78f3dfb53fb70973a: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:12.043128 8484 tablet_bootstrap.cc:492] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a: No bootstrap required, opened a new log
I20260619 04:34:12.043313 8484 ts_tablet_manager.cc:1397] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260619 04:34:12.044003 8486 tablet_bootstrap.cc:492] T acdc482d294d4e8bbb39537a6c6039d3 P 31b8fdefbb5840d78f3dfb53fb70973a: No bootstrap required, opened a new log
I20260619 04:34:12.044239 8486 ts_tablet_manager.cc:1397] T acdc482d294d4e8bbb39537a6c6039d3 P 31b8fdefbb5840d78f3dfb53fb70973a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.005s
I20260619 04:34:12.044109 8484 raft_consensus.cc:348] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:12.044358 8484 raft_consensus.cc:374] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:12.044446 8484 raft_consensus.cc:729] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b72019478929468888fcd697a4d4968a, State: Initialized, Role: FOLLOWER
I20260619 04:34:12.044751 8484 consensus_queue.cc:260] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:12.045217 8486 raft_consensus.cc:348] T acdc482d294d4e8bbb39537a6c6039d3 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:12.045516 8486 raft_consensus.cc:374] T acdc482d294d4e8bbb39537a6c6039d3 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:12.045593 8486 raft_consensus.cc:729] T acdc482d294d4e8bbb39537a6c6039d3 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b8fdefbb5840d78f3dfb53fb70973a, State: Initialized, Role: FOLLOWER
I20260619 04:34:12.045907 8486 consensus_queue.cc:260] T acdc482d294d4e8bbb39537a6c6039d3 P 31b8fdefbb5840d78f3dfb53fb70973a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:12.046257 8485 tablet_bootstrap.cc:492] T acdc482d294d4e8bbb39537a6c6039d3 P eeafa8fcfab041fb8fb588988c4aa694: No bootstrap required, opened a new log
I20260619 04:34:12.046434 8485 ts_tablet_manager.cc:1397] T acdc482d294d4e8bbb39537a6c6039d3 P eeafa8fcfab041fb8fb588988c4aa694: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.000s
I20260619 04:34:12.047164 8485 raft_consensus.cc:348] T acdc482d294d4e8bbb39537a6c6039d3 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:12.047437 8485 raft_consensus.cc:374] T acdc482d294d4e8bbb39537a6c6039d3 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:12.047518 8485 raft_consensus.cc:729] T acdc482d294d4e8bbb39537a6c6039d3 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeafa8fcfab041fb8fb588988c4aa694, State: Initialized, Role: FOLLOWER
I20260619 04:34:12.047751 8485 consensus_queue.cc:260] T acdc482d294d4e8bbb39537a6c6039d3 P eeafa8fcfab041fb8fb588988c4aa694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:12.048245 8485 ts_tablet_manager.cc:1428] T acdc482d294d4e8bbb39537a6c6039d3 P eeafa8fcfab041fb8fb588988c4aa694: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 04:34:12.049440 8486 ts_tablet_manager.cc:1428] T acdc482d294d4e8bbb39537a6c6039d3 P 31b8fdefbb5840d78f3dfb53fb70973a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 04:34:12.049609 8484 ts_tablet_manager.cc:1428] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260619 04:34:12.459059 8489 raft_consensus.cc:482] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:34:12.459266 8489 raft_consensus.cc:504] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:12.460676 8489 leader_election.cc:288] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eeafa8fcfab041fb8fb588988c4aa694 (127.0.97.1:35349), 31b8fdefbb5840d78f3dfb53fb70973a (127.0.97.2:36779)
I20260619 04:34:12.467173 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:12.461049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33573 (local address 127.0.97.1:35349)
0619 04:34:12.461704 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:12.461710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:12.462099 (+ 389us) server_negotiation.cc:407] Connection header received
0619 04:34:12.462278 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:12.462283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:12.462354 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:12.462446 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:12.462955 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.463725 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.465466 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.466266 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.466367 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:12.466475 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:12.466776 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 04:34:12.466919 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":417,"threads_started":1}
I20260619 04:34:12.467183 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:12.461119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44933 (local address 127.0.97.2:36779)
0619 04:34:12.461324 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:12.461330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:12.462256 (+ 926us) server_negotiation.cc:407] Connection header received
0619 04:34:12.462436 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:12.462441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:12.462514 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:12.462602 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:12.463147 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.463854 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.465610 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.466356 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.466473 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:12.466561 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:12.466837 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 04:34:12.466959 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 04:34:12.467219 8567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:12.460923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35349 (local address 127.0.97.3:33573)
0619 04:34:12.461972 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:12.461990 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:12.462133 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:12.462471 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:12.462476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:12.462499 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:12.462805 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:12.462815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.463851 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.463856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:12.465302 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:12.465314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.466439 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.466445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:12.466628 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:12.466651 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 04:34:12.466728 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 04:34:12.466829 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":1,"thread_start_us":782,"threads_started":1}
I20260619 04:34:12.467280 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:12.461001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36779 (local address 127.0.97.3:44933)
0619 04:34:12.462152 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:12.462167 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:12.462303 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:12.462635 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:12.462639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:12.462661 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:12.462995 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:12.463008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.463998 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.464003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:12.465478 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:12.465491 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.466576 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.466580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:12.466688 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:12.466712 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 04:34:12.466801 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 04:34:12.466915 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"thread_start_us":852,"threads_started":1}
I20260619 04:34:12.468053 8271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdc482d294d4e8bbb39537a6c6039d3" candidate_uuid: "b72019478929468888fcd697a4d4968a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" is_pre_election: true
I20260619 04:34:12.468053 8132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdc482d294d4e8bbb39537a6c6039d3" candidate_uuid: "b72019478929468888fcd697a4d4968a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeafa8fcfab041fb8fb588988c4aa694" is_pre_election: true
I20260619 04:34:12.468290 8271 raft_consensus.cc:2454] T acdc482d294d4e8bbb39537a6c6039d3 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b72019478929468888fcd697a4d4968a in term 0.
I20260619 04:34:12.468290 8132 raft_consensus.cc:2454] T acdc482d294d4e8bbb39537a6c6039d3 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b72019478929468888fcd697a4d4968a in term 0.
I20260619 04:34:12.468771 8344 leader_election.cc:302] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b8fdefbb5840d78f3dfb53fb70973a, b72019478929468888fcd697a4d4968a; no voters:
I20260619 04:34:12.469077 8489 raft_consensus.cc:2781] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:34:12.469184 8489 raft_consensus.cc:482] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:34:12.469281 8489 raft_consensus.cc:3037] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:12.471271 8489 raft_consensus.cc:504] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:12.471794 8489 leader_election.cc:288] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a [CANDIDATE]: Term 1 election: Requested vote from peers eeafa8fcfab041fb8fb588988c4aa694 (127.0.97.1:35349), 31b8fdefbb5840d78f3dfb53fb70973a (127.0.97.2:36779)
I20260619 04:34:12.472270 8271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdc482d294d4e8bbb39537a6c6039d3" candidate_uuid: "b72019478929468888fcd697a4d4968a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b8fdefbb5840d78f3dfb53fb70973a"
I20260619 04:34:12.472270 8132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdc482d294d4e8bbb39537a6c6039d3" candidate_uuid: "b72019478929468888fcd697a4d4968a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeafa8fcfab041fb8fb588988c4aa694"
I20260619 04:34:12.472487 8132 raft_consensus.cc:3037] T acdc482d294d4e8bbb39537a6c6039d3 P eeafa8fcfab041fb8fb588988c4aa694 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:12.472487 8271 raft_consensus.cc:3037] T acdc482d294d4e8bbb39537a6c6039d3 P 31b8fdefbb5840d78f3dfb53fb70973a [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:12.475209 8132 raft_consensus.cc:2454] T acdc482d294d4e8bbb39537a6c6039d3 P eeafa8fcfab041fb8fb588988c4aa694 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b72019478929468888fcd697a4d4968a in term 1.
I20260619 04:34:12.475205 8271 raft_consensus.cc:2454] T acdc482d294d4e8bbb39537a6c6039d3 P 31b8fdefbb5840d78f3dfb53fb70973a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b72019478929468888fcd697a4d4968a in term 1.
I20260619 04:34:12.475612 8343 leader_election.cc:302] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 31b8fdefbb5840d78f3dfb53fb70973a, b72019478929468888fcd697a4d4968a, eeafa8fcfab041fb8fb588988c4aa694; no voters:
I20260619 04:34:12.475934 8489 raft_consensus.cc:2781] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:34:12.476305 8489 raft_consensus.cc:686] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a [term 1 LEADER]: Becoming Leader. State: Replica: b72019478929468888fcd697a4d4968a, State: Running, Role: LEADER
I20260619 04:34:12.476688 8489 consensus_queue.cc:237] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } }
I20260619 04:34:12.480631 7974 catalog_manager.cc:5495] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a reported cstate change: term changed from 0 to 1, leader changed from <none> to b72019478929468888fcd697a4d4968a (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b72019478929468888fcd697a4d4968a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b72019478929468888fcd697a4d4968a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38111 } health_report { overall_health: HEALTHY } } }
I20260619 04:34:12.506752 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:12.501408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40646 (local address 127.0.97.3:38111)
0619 04:34:12.501925 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:12.501929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:12.501948 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:34:12.502029 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:12.502032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:12.502106 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:12.502240 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:34:12.502648 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.503419 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.504785 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.505198 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.505284 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:12.505621 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:34:12.506196 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:34:12.506298 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:12.506369 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 04:34:12.506436 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":305,"threads_started":1}
I20260619 04:34:12.520161 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:12.515677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59087 (local address 127.0.97.2:36779)
0619 04:34:12.515842 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:12.515846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:12.516165 (+ 319us) server_negotiation.cc:407] Connection header received
0619 04:34:12.516306 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:12.516309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:12.516381 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:12.516472 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:34:12.516883 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.517567 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.518807 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.519200 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.519286 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:12.519619 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:34:12.519711 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:34:12.519770 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:12.519854 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 04:34:12.519940 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 04:34:12.520217 8585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:12.515534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36779 (local address 127.0.97.3:59087)
0619 04:34:12.516076 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:12.516091 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:12.516213 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:12.516493 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:12.516496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:12.516505 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:34:12.516763 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:12.516770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.517664 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.517668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:12.518691 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:12.518701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.519338 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.519342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:12.519447 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:12.519495 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:34:12.519777 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:34:12.519783 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 04:34:12.519832 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 04:34:12.519950 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":286,"threads_started":1}
I20260619 04:34:12.527477 8271 raft_consensus.cc:1261] T acdc482d294d4e8bbb39537a6c6039d3 P 31b8fdefbb5840d78f3dfb53fb70973a [term 1 FOLLOWER]: Refusing update from remote peer b72019478929468888fcd697a4d4968a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:34:12.527482 8132 raft_consensus.cc:1261] T acdc482d294d4e8bbb39537a6c6039d3 P eeafa8fcfab041fb8fb588988c4aa694 [term 1 FOLLOWER]: Refusing update from remote peer b72019478929468888fcd697a4d4968a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:34:12.528050 8570 consensus_queue.cc:1035] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeafa8fcfab041fb8fb588988c4aa694" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:34:12.528445 8489 consensus_queue.cc:1035] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b8fdefbb5840d78f3dfb53fb70973a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:34:12.552037 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:12.546602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53827 (local address 127.0.97.3:38111)
0619 04:34:12.547002 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:12.547008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:12.547217 (+ 209us) server_negotiation.cc:407] Connection header received
0619 04:34:12.547429 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:12.547434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:12.547507 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:12.547562 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:34:12.548165 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.549017 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.550447 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.550894 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.551016 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:12.551464 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:34:12.551582 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:34:12.551649 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:12.551777 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 04:34:12.551840 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286}
I20260619 04:34:12.552078 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:12.546619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38111 (local address 127.0.97.2:53827)
0619 04:34:12.547132 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:12.547146 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:12.547270 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:12.547754 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:12.547758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:12.547771 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:34:12.548039 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:12.548051 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.549140 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.549146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:12.550326 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:12.550337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.551014 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.551020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:12.551183 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:12.551241 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:34:12.551680 (+ 439us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:34:12.551689 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 04:34:12.551753 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 04:34:12.551851 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":1034,"threads_started":1}
I20260619 04:34:12.587713 7974 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:34:12.587984 7974 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:34:12.590763 7974 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a3b8f71c5ca4979b1aba4f0881f9a74: Sending DeleteTablet for 3 replicas of tablet acdc482d294d4e8bbb39537a6c6039d3
I20260619 04:34:12.591562 8111 tablet_service.cc:1514] Processing DeleteTablet for tablet acdc482d294d4e8bbb39537a6c6039d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:34:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55310
I20260619 04:34:12.591614 8389 tablet_service.cc:1514] Processing DeleteTablet for tablet acdc482d294d4e8bbb39537a6c6039d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:34:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40630
I20260619 04:34:12.591645 8251 tablet_service.cc:1514] Processing DeleteTablet for tablet acdc482d294d4e8bbb39537a6c6039d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:34:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59324
I20260619 04:34:12.592921 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 8046
I20260619 04:34:12.603708 8593 tablet_replica.cc:330] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a: stopping tablet replica
I20260619 04:34:12.604176 8594 tablet_replica.cc:330] T acdc482d294d4e8bbb39537a6c6039d3 P eeafa8fcfab041fb8fb588988c4aa694: stopping tablet replica
I20260619 04:34:12.604491 8594 raft_consensus.cc:2229] T acdc482d294d4e8bbb39537a6c6039d3 P eeafa8fcfab041fb8fb588988c4aa694 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:34:12.604854 8593 raft_consensus.cc:2229] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:34:12.605265 8594 raft_consensus.cc:2258] T acdc482d294d4e8bbb39537a6c6039d3 P eeafa8fcfab041fb8fb588988c4aa694 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:34:12.605856 8593 raft_consensus.cc:2258] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:34:12.607024 8594 ts_tablet_manager.cc:1905] T acdc482d294d4e8bbb39537a6c6039d3 P eeafa8fcfab041fb8fb588988c4aa694: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:12.607719 8593 ts_tablet_manager.cc:1905] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:12.619596 8593 ts_tablet_manager.cc:1918] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:34:12.621388 8594 ts_tablet_manager.cc:1918] T acdc482d294d4e8bbb39537a6c6039d3 P eeafa8fcfab041fb8fb588988c4aa694: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:34:12.621562 8594 log.cc:1196] T acdc482d294d4e8bbb39537a6c6039d3 P eeafa8fcfab041fb8fb588988c4aa694: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-0/wal/wals/acdc482d294d4e8bbb39537a6c6039d3
I20260619 04:34:12.622021 8594 ts_tablet_manager.cc:1939] T acdc482d294d4e8bbb39537a6c6039d3 P eeafa8fcfab041fb8fb588988c4aa694: Deleting consensus metadata
I20260619 04:34:12.623126 8593 log.cc:1196] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/acdc482d294d4e8bbb39537a6c6039d3
I20260619 04:34:12.623636 8593 ts_tablet_manager.cc:1939] T acdc482d294d4e8bbb39537a6c6039d3 P b72019478929468888fcd697a4d4968a: Deleting consensus metadata
I20260619 04:34:12.624274 8595 tablet_replica.cc:330] T acdc482d294d4e8bbb39537a6c6039d3 P 31b8fdefbb5840d78f3dfb53fb70973a: stopping tablet replica
I20260619 04:34:12.624495 8595 raft_consensus.cc:2229] T acdc482d294d4e8bbb39537a6c6039d3 P 31b8fdefbb5840d78f3dfb53fb70973a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:34:12.624778 8595 raft_consensus.cc:2258] T acdc482d294d4e8bbb39537a6c6039d3 P 31b8fdefbb5840d78f3dfb53fb70973a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:34:12.624850 7961 catalog_manager.cc:4841] TS eeafa8fcfab041fb8fb588988c4aa694 (127.0.97.1:35349): tablet acdc482d294d4e8bbb39537a6c6039d3 (table test-table [id=00d4ee4bba0d4e52b284b1f0214af747]) successfully deleted
I20260619 04:34:12.624846 7960 catalog_manager.cc:4841] TS b72019478929468888fcd697a4d4968a (127.0.97.3:38111): tablet acdc482d294d4e8bbb39537a6c6039d3 (table test-table [id=00d4ee4bba0d4e52b284b1f0214af747]) successfully deleted
I20260619 04:34:12.625878 8595 ts_tablet_manager.cc:1905] T acdc482d294d4e8bbb39537a6c6039d3 P 31b8fdefbb5840d78f3dfb53fb70973a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:12.631626 8595 ts_tablet_manager.cc:1918] T acdc482d294d4e8bbb39537a6c6039d3 P 31b8fdefbb5840d78f3dfb53fb70973a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:34:12.631754 8595 log.cc:1196] T acdc482d294d4e8bbb39537a6c6039d3 P 31b8fdefbb5840d78f3dfb53fb70973a: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/acdc482d294d4e8bbb39537a6c6039d3
I20260619 04:34:12.632146 8595 ts_tablet_manager.cc:1939] T acdc482d294d4e8bbb39537a6c6039d3 P 31b8fdefbb5840d78f3dfb53fb70973a: Deleting consensus metadata
I20260619 04:34:12.633131 7962 catalog_manager.cc:4841] TS 31b8fdefbb5840d78f3dfb53fb70973a (127.0.97.2:36779): tablet acdc482d294d4e8bbb39537a6c6039d3 (table test-table [id=00d4ee4bba0d4e52b284b1f0214af747]) successfully deleted
Jun 19 04:34:12 dist-test-slave-n3tq krb5kdc[7930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260619 04:34:12.642267 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:12.594409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55332 (local address 127.0.97.1:35349)
0619 04:34:12.594581 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:12.594588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:12.624344 (+ 29756us) server_negotiation.cc:407] Connection header received
0619 04:34:12.624408 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:12.624414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:12.624517 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:12.624613 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:12.627177 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.628247 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.630316 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.630935 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.631033 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:12.635935 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:12.635953 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:12.635974 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:12.636023 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:12.639596 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:12.640249 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:12.640258 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:12.640262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:12.640431 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:12.640786 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:12.640791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:12.640795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:12.641241 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:12.641533 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:12.641723 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 04:34:12.641937 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 04:34:12.642943 8177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:12.774951 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 8046
I20260619 04:34:12.801882 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 8186
Jun 19 04:34:12 dist-test-slave-n3tq krb5kdc[7930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260619 04:34:12.817466 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:12.802601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59348 (local address 127.0.97.2:36779)
0619 04:34:12.802776 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:12.802780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:12.802796 (+ 16us) server_negotiation.cc:407] Connection header received
0619 04:34:12.802976 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:12.802979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:12.803038 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:12.803116 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:12.804675 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.805373 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.807091 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.807503 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.807593 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:12.812164 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:12.812179 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:12.812193 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:12.812238 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:12.814965 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:12.815550 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:12.815557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:12.815559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:12.815705 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:12.816040 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:12.816044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:12.816046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:12.816352 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:12.816672 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:12.816948 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 04:34:12.817149 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 04:34:12.818030 8316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:12.956934 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 8186
I20260619 04:34:12.987562 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 8325
Jun 19 04:34:12 dist-test-slave-n3tq krb5kdc[7930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260619 04:34:13.002200 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:12.988297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40660 (local address 127.0.97.3:38111)
0619 04:34:12.988497 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:12.988502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:12.988583 (+ 81us) server_negotiation.cc:407] Connection header received
0619 04:34:12.988766 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:12.988770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:12.988829 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:12.988922 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:12.990123 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.990832 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.992032 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:12.992566 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:12.992682 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:12.996842 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:12.996854 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:12.996869 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:12.996900 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:12.999617 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:13.000229 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:13.000236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:13.000238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:13.000361 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:13.000725 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:13.000729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:13.000731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:13.001054 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:13.001329 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:13.001635 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 04:34:13.001866 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 04:34:13.002877 8455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:13.124607 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 8325
I20260619 04:34:13.142303 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 7943
I20260619 04:34:13.143426 8015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:13.254638 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 7943
2026-06-19T04:34:13Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5740 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:34:13 dist-test-slave-n3tq krb5kdc[8603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:34:13 dist-test-slave-n3tq krb5kdc[8603](info): set up 2 sockets
Jun 19 04:34:13 dist-test-slave-n3tq krb5kdc[8603](info): commencing operation
krb5kdc: starting...
W20260619 04:34:14.261442 388 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.005s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:34:14 dist-test-slave-n3tq krb5kdc[8603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:34:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:34:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:34:14.464102 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41423
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:34:14.781834 8616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:14.782132 8616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:14.782265 8616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:14.789314 8616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:34:14.789425 8616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:14.789492 8616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:14.789563 8616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:34:14.789626 8616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:34:14.799401 8616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:14.800691 8616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:14.801994 8616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:14.813043 8621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:14.813007 8622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:14.813082 8624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:14.814165 8616 server_base.cc:1034] running on GCE node
I20260619 04:34:14.814878 8616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:14.816182 8616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:14.817376 8616 hybrid_clock.cc:648] HybridClock initialized: now 1781843654817327 us; error 39 us; skew 500 ppm
I20260619 04:34:14.817696 8616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T04:34:14Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1627 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:34:14 dist-test-slave-n3tq krb5kdc[8630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:34:14 dist-test-slave-n3tq krb5kdc[8630](info): set up 2 sockets
Jun 19 04:34:14 dist-test-slave-n3tq krb5kdc[8630](info): commencing operation
krb5kdc: starting...
W20260619 04:34:15.893046 388 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:34:16 dist-test-slave-n3tq krb5kdc[8630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:34:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:34:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:34:16.086032 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43013
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:34:16.390414 8643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:16.390718 8643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:16.390848 8643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:16.397912 8643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:34:16.398030 8643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:16.398109 8643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:16.398185 8643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:34:16.398244 8643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:34:16.407850 8643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:16.409204 8643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:16.410436 8643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:16.421299 8648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:16.421231 8651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:16.421270 8649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:16.422241 8643 server_base.cc:1034] running on GCE node
I20260619 04:34:16.422977 8643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:16.424275 8643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:16.425472 8643 hybrid_clock.cc:648] HybridClock initialized: now 1781843656425425 us; error 34 us; skew 500 ppm
I20260619 04:34:16.425786 8643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:16 dist-test-slave-n3tq krb5kdc[8630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:16.428566 8643 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:34:16.429888 8643 webserver.cc:460] Webserver started at http://127.0.97.62:43581/ using document root <none> and password file <none>
I20260619 04:34:16.430501 8643 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:16.430616 8643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:16.430895 8643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:16.433606 8643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "511df68df5114cceaaf9a3bf327be570"
format_stamp: "Formatted at 2026-06-19 04:34:16 on dist-test-slave-n3tq"
server_key: "9d9654bd5506eed02df97a6873d9cb10"
server_key_iv: "d9804592ad291c48bc524ac5b206e594"
server_key_version: "encryptionkey@0"
I20260619 04:34:16.434319 8643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "511df68df5114cceaaf9a3bf327be570"
format_stamp: "Formatted at 2026-06-19 04:34:16 on dist-test-slave-n3tq"
server_key: "9d9654bd5506eed02df97a6873d9cb10"
server_key_iv: "d9804592ad291c48bc524ac5b206e594"
server_key_version: "encryptionkey@0"
I20260619 04:34:16.439054 8643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 04:34:16.442139 8658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:16.443320 8643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 04:34:16.443470 8643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "511df68df5114cceaaf9a3bf327be570"
format_stamp: "Formatted at 2026-06-19 04:34:16 on dist-test-slave-n3tq"
server_key: "9d9654bd5506eed02df97a6873d9cb10"
server_key_iv: "d9804592ad291c48bc524ac5b206e594"
server_key_version: "encryptionkey@0"
I20260619 04:34:16.443631 8643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:16.459966 8643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:16.463493 8643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:16.463725 8643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:16.479945 8643 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43013
I20260619 04:34:16.479974 8709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43013 every 8 connection(s)
I20260619 04:34:16.481204 8643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:34:16.481890 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 8643
I20260619 04:34:16.482091 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:34:16.482805 388 external_mini_cluster.cc:1428] Setting key b7bc7e977f2cc4fa07d3504259f3e13a
I20260619 04:34:16.485361 8710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 04:34:16 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:34:16.497733 8710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570: Bootstrap starting.
I20260619 04:34:16.501011 8710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:16.501979 8710 log.cc:824] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570: Log is configured to *not* fsync() on all Append() calls
I20260619 04:34:16.504668 8710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570: No bootstrap required, opened a new log
I20260619 04:34:16.504922 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:16.484619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60362 (local address 127.0.97.62:43013)
0619 04:34:16.485364 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:16.485378 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:16.485428 (+ 50us) server_negotiation.cc:407] Connection header received
0619 04:34:16.487584 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:16.487611 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:16.488094 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:16.488538 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:16.489734 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:16.491712 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:16.492941 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:16.493900 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:16.494070 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:16.498077 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:16.498085 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:16.498102 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:16.498137 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:16.501135 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:16.501780 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:16.501786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:16.501792 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:16.501951 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:16.502333 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:16.502337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:16.502339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:16.502848 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:16.503112 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:16.503545 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 04:34:16.503922 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":300,"threads_started":1}
I20260619 04:34:16.512785 8710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511df68df5114cceaaf9a3bf327be570" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } }
I20260619 04:34:16.513131 8710 raft_consensus.cc:374] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:16.513222 8710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 511df68df5114cceaaf9a3bf327be570, State: Initialized, Role: FOLLOWER
I20260619 04:34:16.513749 8710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511df68df5114cceaaf9a3bf327be570" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } }
I20260619 04:34:16.513957 8710 raft_consensus.cc:388] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:34:16.514077 8710 raft_consensus.cc:482] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:34:16.514227 8710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:16.516290 8710 raft_consensus.cc:504] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511df68df5114cceaaf9a3bf327be570" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } }
I20260619 04:34:16.516880 8710 leader_election.cc:302] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 511df68df5114cceaaf9a3bf327be570; no voters:
I20260619 04:34:16.517491 8710 leader_election.cc:288] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:34:16.517565 8715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:34:16.518321 8715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [term 1 LEADER]: Becoming Leader. State: Replica: 511df68df5114cceaaf9a3bf327be570, State: Running, Role: LEADER
I20260619 04:34:16.518878 8715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511df68df5114cceaaf9a3bf327be570" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } }
I20260619 04:34:16.519500 8710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:34:16.521369 8716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "511df68df5114cceaaf9a3bf327be570" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511df68df5114cceaaf9a3bf327be570" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } } }
I20260619 04:34:16.521847 8716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [sys.catalog]: This master's current role is: LEADER
I20260619 04:34:16.521727 8717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 511df68df5114cceaaf9a3bf327be570. Latest consensus state: current_term: 1 leader_uuid: "511df68df5114cceaaf9a3bf327be570" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511df68df5114cceaaf9a3bf327be570" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } } }
I20260619 04:34:16.521996 8717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [sys.catalog]: This master's current role is: LEADER
I20260619 04:34:16.524096 8723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:34:16.529291 8723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:34:16.537314 8723 catalog_manager.cc:1347] Generated new cluster ID: 962d5cfa812c4e359950c2f569734c3f
I20260619 04:34:16.537451 8723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:34:16.573323 8723 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:34:16.574643 8723 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:34:16.581858 8723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570: Generated new TSK 0
I20260619 04:34:16.582692 8723 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 04:34:16.640167 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43013
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:34:16.947373 8736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:16.947659 8736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:16.947777 8736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:16.954908 8736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:16.955013 8736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:16.955123 8736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:34:16.964692 8736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:16.965871 8736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:16.967231 8736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:16.979195 8741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:16.979264 8742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:16.979298 8744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:16.980291 8736 server_base.cc:1034] running on GCE node
I20260619 04:34:16.980852 8736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:16.981739 8736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:16.982934 8736 hybrid_clock.cc:648] HybridClock initialized: now 1781843656982884 us; error 45 us; skew 500 ppm
I20260619 04:34:16.983217 8736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:16 dist-test-slave-n3tq krb5kdc[8630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:16.986142 8736 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:34:16.987444 8736 webserver.cc:460] Webserver started at http://127.0.97.1:35747/ using document root <none> and password file <none>
I20260619 04:34:16.988122 8736 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:16.988232 8736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:16.988543 8736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:16.991281 8736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "b9c6cd3f8c2b4acc8a3016ed37218f38"
format_stamp: "Formatted at 2026-06-19 04:34:16 on dist-test-slave-n3tq"
server_key: "2ea46bfd2a4eb8168db494bbdd432608"
server_key_iv: "3f2f07f00d050914fe73752edf2537b1"
server_key_version: "encryptionkey@0"
I20260619 04:34:16.992030 8736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b9c6cd3f8c2b4acc8a3016ed37218f38"
format_stamp: "Formatted at 2026-06-19 04:34:16 on dist-test-slave-n3tq"
server_key: "2ea46bfd2a4eb8168db494bbdd432608"
server_key_iv: "3f2f07f00d050914fe73752edf2537b1"
server_key_version: "encryptionkey@0"
I20260619 04:34:16.996963 8736 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 04:34:17.000388 8751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:17.001758 8736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:34:17.001924 8736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "b9c6cd3f8c2b4acc8a3016ed37218f38"
format_stamp: "Formatted at 2026-06-19 04:34:16 on dist-test-slave-n3tq"
server_key: "2ea46bfd2a4eb8168db494bbdd432608"
server_key_iv: "3f2f07f00d050914fe73752edf2537b1"
server_key_version: "encryptionkey@0"
I20260619 04:34:17.002116 8736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:17.027830 8736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:17.031942 8736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:17.032198 8736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:17.033178 8736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:34:17.034458 8736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:17.034530 8736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:17.034646 8736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:17.034696 8736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:17.063813 8736 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34419
I20260619 04:34:17.063856 8863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34419 every 8 connection(s)
I20260619 04:34:17.065018 8736 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:34:17.069360 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 8736
I20260619 04:34:17.069578 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:34:17.070302 388 external_mini_cluster.cc:1428] Setting key 048e41d70064923ca79ebe91f7690c22
Jun 19 04:34:17 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:34:17.084075 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:17.067319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33795 (local address 127.0.97.62:43013)
0619 04:34:17.067725 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:17.067730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:17.068598 (+ 868us) server_negotiation.cc:407] Connection header received
0619 04:34:17.071040 (+ 2442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:17.071046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:17.071126 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:17.071216 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:17.073025 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:17.073695 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:17.075157 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:17.075547 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:17.075652 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:17.078815 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:17.078820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:17.078823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:17.078853 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:17.081237 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:17.081915 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:17.081919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:17.081922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:17.082056 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:17.082445 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:17.082448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:17.082450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:17.082704 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:17.082826 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:17.083473 (+ 647us) server_negotiation.cc:299] Negotiation successful
0619 04:34:17.083673 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":222,"threads_started":1}
I20260619 04:34:17.085016 8867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:17.067765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:33795)
0619 04:34:17.068419 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:17.068492 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:17.070732 (+ 2240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:17.071475 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:17.071485 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:17.072191 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:17.072867 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:17.072884 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:17.073835 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:17.073839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:17.075023 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:17.075033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:17.075754 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:17.075761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:17.075971 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:17.076579 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:17.076601 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:17.078610 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:17.081365 (+ 2755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:17.081374 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:17.081392 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:17.081789 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:17.082156 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:17.082161 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:17.082165 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:17.082333 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:17.082827 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:17.082835 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:17.083137 (+ 302us) client_negotiation.cc:769] Sending connection context
0619 04:34:17.083361 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 04:34:17.083870 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":248,"threads_started":1}
I20260619 04:34:17.086730 8864 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43013
I20260619 04:34:17.087011 8864 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:17.087657 8864 heartbeater.cc:511] Master 127.0.97.62:43013 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 04:34:17.089764 8675 ts_manager.cc:194] Registered new tserver with Master: b9c6cd3f8c2b4acc8a3016ed37218f38 (127.0.97.1:34419)
I20260619 04:34:17.091228 8675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 04:34:17.114820 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43013
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:34:17.419426 8870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:17.419700 8870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:17.419839 8870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:17.427129 8870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:17.427253 8870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:17.427405 8870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:34:17.436962 8870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:17.438284 8870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:17.439539 8870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:17.451598 8875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:17.451519 8878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:17.451555 8876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:17.452674 8870 server_base.cc:1034] running on GCE node
I20260619 04:34:17.453203 8870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:17.454043 8870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:17.455231 8870 hybrid_clock.cc:648] HybridClock initialized: now 1781843657455189 us; error 32 us; skew 500 ppm
I20260619 04:34:17.455510 8870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:17 dist-test-slave-n3tq krb5kdc[8630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:17.458400 8870 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:34:17.459635 8870 webserver.cc:460] Webserver started at http://127.0.97.2:45135/ using document root <none> and password file <none>
I20260619 04:34:17.460258 8870 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:17.460373 8870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:17.460749 8870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:17.463449 8870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "3e03212f55134928a296f84c8d124d7f"
format_stamp: "Formatted at 2026-06-19 04:34:17 on dist-test-slave-n3tq"
server_key: "eb4df4077ecba0504f34963da6549caa"
server_key_iv: "b1c158822941a8ff289dd4bd6ff348ed"
server_key_version: "encryptionkey@0"
I20260619 04:34:17.464175 8870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3e03212f55134928a296f84c8d124d7f"
format_stamp: "Formatted at 2026-06-19 04:34:17 on dist-test-slave-n3tq"
server_key: "eb4df4077ecba0504f34963da6549caa"
server_key_iv: "b1c158822941a8ff289dd4bd6ff348ed"
server_key_version: "encryptionkey@0"
I20260619 04:34:17.468943 8870 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 04:34:17.472111 8885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:17.473417 8870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 04:34:17.473559 8870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "3e03212f55134928a296f84c8d124d7f"
format_stamp: "Formatted at 2026-06-19 04:34:17 on dist-test-slave-n3tq"
server_key: "eb4df4077ecba0504f34963da6549caa"
server_key_iv: "b1c158822941a8ff289dd4bd6ff348ed"
server_key_version: "encryptionkey@0"
I20260619 04:34:17.473735 8870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:17.500094 8870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:17.504326 8870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:17.504622 8870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:17.505620 8870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:34:17.506990 8870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:17.507064 8870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:17.507170 8870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:17.507220 8870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:17.535637 8870 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41837
I20260619 04:34:17.535684 8997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41837 every 8 connection(s)
I20260619 04:34:17.536897 8870 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:34:17.542233 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 8870
I20260619 04:34:17.542412 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:34:17.542805 388 external_mini_cluster.cc:1428] Setting key c167de2d54e18a7a651ebc178c7eb680
Jun 19 04:34:17 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:34:17.556275 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:17.539322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58695 (local address 127.0.97.62:43013)
0619 04:34:17.539494 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:17.539499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:17.540620 (+ 1121us) server_negotiation.cc:407] Connection header received
0619 04:34:17.542393 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:17.542399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:17.542485 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:17.542578 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:17.544188 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:17.545036 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:17.546618 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:17.547013 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:17.547115 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:17.550451 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:17.550457 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:17.550462 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:17.550509 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:17.553447 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:17.554150 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:17.554155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:17.554158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:17.554304 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:17.554658 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:17.554663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:17.554665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:17.554974 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:17.555109 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:17.555760 (+ 651us) server_negotiation.cc:299] Negotiation successful
0619 04:34:17.555957 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 04:34:17.557229 9000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:17.539796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:58695)
0619 04:34:17.540443 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:17.540505 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:17.542124 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:17.542802 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:17.542810 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:17.543435 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:17.544024 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:17.544041 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:17.545178 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:17.545183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:17.546488 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:17.546503 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:17.547213 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:17.547220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:17.547459 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:17.548100 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:17.548121 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:17.550215 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:17.553589 (+ 3374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:17.553596 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:17.553608 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:17.554043 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:17.554400 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:17.554403 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:17.554405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:17.554569 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:17.555117 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:17.555123 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:17.555426 (+ 303us) client_negotiation.cc:769] Sending connection context
0619 04:34:17.555655 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 04:34:17.556044 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":252,"threads_started":1}
I20260619 04:34:17.558534 8998 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43013
I20260619 04:34:17.558840 8998 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:17.559682 8998 heartbeater.cc:511] Master 127.0.97.62:43013 requested a full tablet report, sending...
I20260619 04:34:17.561247 8675 ts_manager.cc:194] Registered new tserver with Master: 3e03212f55134928a296f84c8d124d7f (127.0.97.2:41837)
I20260619 04:34:17.562022 8675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58695
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 04:34:17.587735 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43013
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:34:17.893426 9003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:17.893718 9003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:17.893839 9003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:17.900897 9003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:17.901031 9003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:17.901175 9003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:34:17.910764 9003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:17.912030 9003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:17.913309 9003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:17.925539 9008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:17.925562 9009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:17.925491 9011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:17.926224 9003 server_base.cc:1034] running on GCE node
I20260619 04:34:17.926782 9003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:17.927657 9003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:17.928836 9003 hybrid_clock.cc:648] HybridClock initialized: now 1781843657928791 us; error 39 us; skew 500 ppm
I20260619 04:34:17.929121 9003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:17 dist-test-slave-n3tq krb5kdc[8630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:17.931933 9003 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:34:17.933282 9003 webserver.cc:460] Webserver started at http://127.0.97.3:41949/ using document root <none> and password file <none>
I20260619 04:34:17.933920 9003 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:17.934033 9003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:17.934309 9003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:17.937152 9003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "d37e49bfab544fa9bb4ecd68cc27c837"
format_stamp: "Formatted at 2026-06-19 04:34:17 on dist-test-slave-n3tq"
server_key: "07c72010832c3af83c21dd568dc7d5d8"
server_key_iv: "6a59f8e3620c069eda48034da0f1fc61"
server_key_version: "encryptionkey@0"
I20260619 04:34:17.937863 9003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d37e49bfab544fa9bb4ecd68cc27c837"
format_stamp: "Formatted at 2026-06-19 04:34:17 on dist-test-slave-n3tq"
server_key: "07c72010832c3af83c21dd568dc7d5d8"
server_key_iv: "6a59f8e3620c069eda48034da0f1fc61"
server_key_version: "encryptionkey@0"
I20260619 04:34:17.942682 9003 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 04:34:17.945966 9018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:17.947248 9003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260619 04:34:17.947386 9003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "d37e49bfab544fa9bb4ecd68cc27c837"
format_stamp: "Formatted at 2026-06-19 04:34:17 on dist-test-slave-n3tq"
server_key: "07c72010832c3af83c21dd568dc7d5d8"
server_key_iv: "6a59f8e3620c069eda48034da0f1fc61"
server_key_version: "encryptionkey@0"
I20260619 04:34:17.947588 9003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:17.970314 9003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:17.974159 9003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:17.974403 9003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:17.975382 9003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:34:17.976727 9003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:17.976809 9003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:17.976909 9003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:17.976958 9003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:18.005313 9003 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38809
I20260619 04:34:18.005371 9130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38809 every 8 connection(s)
I20260619 04:34:18.006604 9003 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:34:18.016119 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 9003
I20260619 04:34:18.016281 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:34:18.016747 388 external_mini_cluster.cc:1428] Setting key 2ded0a3aa90610d2160bf77ca7edfff2
Jun 19 04:34:18 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:34:18.025986 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:18.008973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42875 (local address 127.0.97.62:43013)
0619 04:34:18.009130 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:18.009134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:18.010285 (+ 1151us) server_negotiation.cc:407] Connection header received
0619 04:34:18.012064 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:18.012070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:18.012152 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:18.012238 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:18.013730 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:18.014427 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:18.016059 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:18.016572 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:18.016676 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:18.020837 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:18.020842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:18.020845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:18.020884 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:18.023353 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:18.024072 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:18.024076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:18.024078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:18.024202 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:18.024597 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:18.024600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:18.024602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:18.024873 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:18.024983 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:18.025579 (+ 596us) server_negotiation.cc:299] Negotiation successful
0619 04:34:18.025732 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 04:34:18.026916 9133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:18.009438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:42875)
0619 04:34:18.010124 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:18.010160 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:18.011789 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:18.012505 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:18.012518 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:18.013024 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:18.013563 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:18.013580 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:18.014600 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:18.014606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:18.015894 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:18.015909 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:18.016827 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:18.016835 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:18.017126 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:18.017820 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:18.017843 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:18.020548 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:18.023478 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:18.023485 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:18.023506 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:18.023982 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:18.024292 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:18.024295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:18.024297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:18.024490 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:18.024966 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:18.024972 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:18.025281 (+ 309us) client_negotiation.cc:769] Sending connection context
0619 04:34:18.025492 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 04:34:18.025871 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":264,"threads_started":1}
I20260619 04:34:18.028200 9131 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43013
I20260619 04:34:18.028477 9131 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:18.029150 9131 heartbeater.cc:511] Master 127.0.97.62:43013 requested a full tablet report, sending...
I20260619 04:34:18.030447 8675 ts_manager.cc:194] Registered new tserver with Master: d37e49bfab544fa9bb4ecd68cc27c837 (127.0.97.3:38809)
I20260619 04:34:18.031318 8675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42875
I20260619 04:34:18.032128 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260619 04:34:18.094146 8864 heartbeater.cc:503] Master 127.0.97.62:43013 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260619 04:34:18.152407 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 8736
Jun 19 04:34:18 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 04:34:18.167552 9141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:18.153529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42870 (local address 127.0.97.1:34419)
0619 04:34:18.154123 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:18.154139 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:18.154166 (+ 27us) server_negotiation.cc:407] Connection header received
0619 04:34:18.154228 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:18.154234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:18.154442 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:18.154601 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:18.155798 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:18.156855 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:18.158101 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:18.158661 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:18.158765 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:18.162599 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:18.162614 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:18.162634 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:18.162679 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:18.165334 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:18.165902 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:18.165909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:18.165911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:18.166033 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:18.166282 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:18.166285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:18.166287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:18.166594 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:18.166798 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:18.167025 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 04:34:18.167206 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":308,"threads_started":1}
I20260619 04:34:18.168773 8859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:18.248509 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 8736
I20260619 04:34:18.261572 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 8870
Jun 19 04:34:18 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 04:34:18.276810 9143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:18.262316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60960 (local address 127.0.97.2:41837)
0619 04:34:18.262852 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:18.262860 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:18.262886 (+ 26us) server_negotiation.cc:407] Connection header received
0619 04:34:18.262943 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:18.262950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:18.263162 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:18.263302 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:18.264410 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:18.265417 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:18.266654 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:18.267191 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:18.267291 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:18.271484 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:18.271497 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:18.271520 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:18.271554 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:18.274076 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:18.274718 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:18.274725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:18.274728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:18.274845 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:18.275205 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:18.275209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:18.275211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:18.275557 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:18.275836 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:18.276140 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 04:34:18.276426 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":268,"threads_started":1}
I20260619 04:34:18.278127 8993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:18.358647 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 8870
I20260619 04:34:18.370852 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 9003
Jun 19 04:34:18 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 04:34:18.386243 9145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:18.371925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47194 (local address 127.0.97.3:38809)
0619 04:34:18.372564 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:18.372571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:18.372599 (+ 28us) server_negotiation.cc:407] Connection header received
0619 04:34:18.372657 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:18.372664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:18.372871 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:18.372997 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:18.374227 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:18.375196 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:18.376449 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:18.377157 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:18.377256 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:18.381310 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:18.381322 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:18.381338 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:18.381375 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:18.383844 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:18.384439 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:18.384446 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:18.384479 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:18.384617 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:18.384915 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:18.384918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:18.384920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:18.385256 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:18.385481 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:18.385692 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 04:34:18.385887 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":315,"threads_started":1}
I20260619 04:34:18.387449 9126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:18.467795 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 9003
I20260619 04:34:18.479993 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 8643
I20260619 04:34:18.481150 8705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:18.579885 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 8643
I20260619 04:34:18.596055 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43013
--webserver_interface=127.0.97.62
--webserver_port=43581
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:34:18.907591 9148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:18.907897 9148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:18.908034 9148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:18.915159 9148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:34:18.915272 9148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:18.915350 9148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:18.915431 9148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:34:18.915493 9148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:34:18.925240 9148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:18.926532 9148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:18.927789 9148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:18.938715 9153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:18.938710 9154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:18.938660 9156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:18.939565 9148 server_base.cc:1034] running on GCE node
I20260619 04:34:18.940351 9148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:18.941695 9148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:18.942883 9148 hybrid_clock.cc:648] HybridClock initialized: now 1781843658942842 us; error 33 us; skew 500 ppm
I20260619 04:34:18.943224 9148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:18 dist-test-slave-n3tq krb5kdc[8630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:18.946091 9148 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:34:18.947387 9148 webserver.cc:460] Webserver started at http://127.0.97.62:43581/ using document root <none> and password file <none>
I20260619 04:34:18.948016 9148 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:18.948133 9148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:18.953204 9148 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:34:18.955871 9163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:18.957159 9148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:34:18.957293 9148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "511df68df5114cceaaf9a3bf327be570"
format_stamp: "Formatted at 2026-06-19 04:34:16 on dist-test-slave-n3tq"
server_key: "9d9654bd5506eed02df97a6873d9cb10"
server_key_iv: "d9804592ad291c48bc524ac5b206e594"
server_key_version: "encryptionkey@0"
I20260619 04:34:18.958154 9148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:18.974259 9148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:18.978276 9148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:18.978508 9148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:18.994963 9148 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43013
I20260619 04:34:18.994975 9214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43013 every 8 connection(s)
I20260619 04:34:18.996417 9148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:34:19.002971 9215 sys_catalog.cc:263] Verifying existing consensus state
I20260619 04:34:19.004947 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 9148
I20260619 04:34:19.005646 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:34419
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:43013
--webserver_port=35747
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:34:19.006399 9215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570: Bootstrap starting.
I20260619 04:34:19.025390 9215 log.cc:824] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570: Log is configured to *not* fsync() on all Append() calls
I20260619 04:34:19.034121 9215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 04:34:19.034597 9215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570: Bootstrap complete.
I20260619 04:34:19.043203 9215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511df68df5114cceaaf9a3bf327be570" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } }
I20260619 04:34:19.044008 9215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 511df68df5114cceaaf9a3bf327be570, State: Initialized, Role: FOLLOWER
I20260619 04:34:19.044597 9215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511df68df5114cceaaf9a3bf327be570" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } }
I20260619 04:34:19.044811 9215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:34:19.044910 9215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:34:19.045066 9215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [term 1 FOLLOWER]: Advancing to term 2
I20260619 04:34:19.048674 9215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511df68df5114cceaaf9a3bf327be570" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } }
I20260619 04:34:19.049118 9215 leader_election.cc:302] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 511df68df5114cceaaf9a3bf327be570; no voters:
I20260619 04:34:19.049711 9215 leader_election.cc:288] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 04:34:19.049840 9219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 04:34:19.050828 9219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [term 2 LEADER]: Becoming Leader. State: Replica: 511df68df5114cceaaf9a3bf327be570, State: Running, Role: LEADER
I20260619 04:34:19.051407 9219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511df68df5114cceaaf9a3bf327be570" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } }
I20260619 04:34:19.051762 9215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:34:19.053936 9221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 511df68df5114cceaaf9a3bf327be570. Latest consensus state: current_term: 2 leader_uuid: "511df68df5114cceaaf9a3bf327be570" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511df68df5114cceaaf9a3bf327be570" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } } }
I20260619 04:34:19.054280 9221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [sys.catalog]: This master's current role is: LEADER
I20260619 04:34:19.054677 9220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "511df68df5114cceaaf9a3bf327be570" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511df68df5114cceaaf9a3bf327be570" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } } }
I20260619 04:34:19.055016 9220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570 [sys.catalog]: This master's current role is: LEADER
I20260619 04:34:19.066143 9228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:34:19.071532 9228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:34:19.072916 9228 catalog_manager.cc:1259] Loaded cluster ID: 962d5cfa812c4e359950c2f569734c3f
I20260619 04:34:19.073076 9228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:34:19.076907 9228 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:34:19.078271 9228 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 511df68df5114cceaaf9a3bf327be570: Loaded TSK: 0
I20260619 04:34:19.079785 9228 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 04:34:19.327239 9217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:19.327526 9217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:19.327652 9217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:19.335037 9217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:19.335155 9217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:19.335302 9217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:34:19.345141 9217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:34419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35747
--tserver_master_addrs=127.0.97.62:43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:19.346427 9217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:19.347666 9217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:19.360180 9241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:19.360209 9242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:19.360817 9244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:19.361418 9217 server_base.cc:1034] running on GCE node
I20260619 04:34:19.362033 9217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:19.363011 9217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:19.364219 9217 hybrid_clock.cc:648] HybridClock initialized: now 1781843659364164 us; error 52 us; skew 500 ppm
I20260619 04:34:19.364567 9217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:19 dist-test-slave-n3tq krb5kdc[8630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843659, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:19.367668 9217 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 04:34:19.369181 9217 webserver.cc:460] Webserver started at http://127.0.97.1:35747/ using document root <none> and password file <none>
I20260619 04:34:19.369834 9217 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:19.369967 9217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:19.375149 9217 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 04:34:19.377732 9251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:19.379014 9217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:34:19.379138 9217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "b9c6cd3f8c2b4acc8a3016ed37218f38"
format_stamp: "Formatted at 2026-06-19 04:34:16 on dist-test-slave-n3tq"
server_key: "2ea46bfd2a4eb8168db494bbdd432608"
server_key_iv: "3f2f07f00d050914fe73752edf2537b1"
server_key_version: "encryptionkey@0"
I20260619 04:34:19.379992 9217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:19.417289 9217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:19.421077 9217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:19.421315 9217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:19.422245 9217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:34:19.423521 9217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:19.423614 9217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:19.423713 9217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:19.423785 9217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:19.452728 9217 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34419
I20260619 04:34:19.452783 9363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34419 every 8 connection(s)
I20260619 04:34:19.454087 9217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:34:19.455715 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 9217
I20260619 04:34:19.456229 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41837
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:43013
--webserver_port=45135
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:34:19 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:19 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:19.472536 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:19.456972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:47575)
0619 04:34:19.457768 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:19.457822 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:19.459801 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:19.461253 (+ 1452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:19.461263 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:19.461874 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:19.462652 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:19.462675 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.464863 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.464866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:19.466133 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:19.466145 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.467404 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.467409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:19.467584 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:19.468185 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:19.468206 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:19.471611 (+ 3405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":295,"threads_started":1}
W20260619 04:34:19.473145 9364 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43013 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:43013: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
W20260619 04:34:19.473206 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:19.458298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47575 (local address 127.0.97.62:43013)
0619 04:34:19.458853 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:19.458863 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:19.458902 (+ 39us) server_negotiation.cc:407] Connection header received
0619 04:34:19.460307 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:19.460320 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:19.460711 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:19.461071 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:19.462872 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.464727 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.466261 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.467229 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.467383 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:19.472043 (+ 4660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47575: BlockingRecv error: recv got EOF from 127.0.97.1:47575 (error 108)
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":220,"threads_started":1}
Jun 19 04:34:19 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:19 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:19.480669 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:19.473660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:55689)
0619 04:34:19.473873 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:19.473889 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:19.473985 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:19.474297 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:19.474300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:19.474557 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:19.474680 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:19.474687 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.475649 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.475653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:19.476542 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:19.476553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.477159 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.477164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:19.477311 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:19.477853 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:19.477868 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:19.480365 (+ 2497us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117}
W20260619 04:34:19.480877 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:19.473858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55689 (local address 127.0.97.62:43013)
0619 04:34:19.474035 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:19.474041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:19.474059 (+ 18us) server_negotiation.cc:407] Connection header received
0619 04:34:19.474108 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:19.474114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:19.474183 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:19.474283 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:19.474828 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.475517 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.476656 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.477037 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.477135 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:19.480581 (+ 3446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55689: BlockingRecv error: recv got EOF from 127.0.97.1:55689 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 04:34:19 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:19 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:19.488144 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:19.481279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:58973)
0619 04:34:19.481413 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:19.481430 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:19.481522 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:19.481888 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:19.481893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:19.482148 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:19.482271 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:19.482278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.483192 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.483212 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:19.484194 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:19.484205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.484879 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.484884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:19.485032 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:19.485528 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:19.485542 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:19.487936 (+ 2394us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 04:34:19.488368 9364 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 04:34:19.488541 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:19.481527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58973 (local address 127.0.97.62:43013)
0619 04:34:19.481667 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:19.481671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:19.481683 (+ 12us) server_negotiation.cc:407] Connection header received
0619 04:34:19.481715 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:19.481718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:19.481773 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:19.481847 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:19.482392 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.483073 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.484326 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.484760 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.484858 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:19.488155 (+ 3297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58973: BlockingRecv error: recv got EOF from 127.0.97.1:58973 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 04:34:19.767259 9368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:19.767510 9368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:19.767598 9368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:19.774523 9368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:19.774621 9368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:19.774729 9368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:34:19.784096 9368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45135
--tserver_master_addrs=127.0.97.62:43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:19.785298 9368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:19.786505 9368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:19.798427 9374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:19.798496 9373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:19.798496 9376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:19.799288 9368 server_base.cc:1034] running on GCE node
I20260619 04:34:19.799815 9368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:19.800725 9368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:19.801915 9368 hybrid_clock.cc:648] HybridClock initialized: now 1781843659801875 us; error 32 us; skew 500 ppm
I20260619 04:34:19.802160 9368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:19 dist-test-slave-n3tq krb5kdc[8630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843659, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:19.804885 9368 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 04:34:19.806123 9368 webserver.cc:460] Webserver started at http://127.0.97.2:45135/ using document root <none> and password file <none>
I20260619 04:34:19.806746 9368 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:19.806870 9368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:19.811811 9368 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:34:19.814440 9383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:19.815717 9368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 04:34:19.815871 9368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "3e03212f55134928a296f84c8d124d7f"
format_stamp: "Formatted at 2026-06-19 04:34:17 on dist-test-slave-n3tq"
server_key: "eb4df4077ecba0504f34963da6549caa"
server_key_iv: "b1c158822941a8ff289dd4bd6ff348ed"
server_key_version: "encryptionkey@0"
I20260619 04:34:19.816844 9368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:19.832746 9368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:19.836603 9368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:19.836872 9368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:19.837777 9368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:34:19.839002 9368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:19.839093 9368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:19.839195 9368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:19.839260 9368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:19.867947 9368 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41837
I20260619 04:34:19.868032 9495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41837 every 8 connection(s)
I20260619 04:34:19.869351 9368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:34:19.878345 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 9368
I20260619 04:34:19.878999 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:38809
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:43013
--webserver_port=41949
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:34:19 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:19 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 04:34:19.885008 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:19.871742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52067 (local address 127.0.97.62:43013)
0619 04:34:19.871902 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:19.871906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:19.873045 (+ 1139us) server_negotiation.cc:407] Connection header received
0619 04:34:19.874769 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:19.874774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:19.874849 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:19.874933 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:19.876386 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.877121 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.878950 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.879444 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.879546 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:19.884824 (+ 5278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52067: BlockingRecv error: recv got EOF from 127.0.97.2:52067 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 04:34:19.885856 9498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:19.872195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:52067)
0619 04:34:19.872889 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:19.872930 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:19.874515 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:19.875166 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:19.875174 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:19.875664 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:19.876215 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:19.876230 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.877263 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.877267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:19.878793 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:19.878806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.879662 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.879669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:19.879941 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:19.880680 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:19.880707 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:19.884671 (+ 3964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":300,"threads_started":1}
W20260619 04:34:19.886472 9496 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43013 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:43013: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 19 04:34:19 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:19 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:19.893949 9498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:19.887008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:46017)
0619 04:34:19.887168 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:19.887186 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:19.887274 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:19.887558 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:19.887561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:19.887817 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:19.887933 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:19.887939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.889124 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.889128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:19.889976 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:19.889985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.890756 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.890759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:19.890869 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:19.891373 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:19.891388 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:19.893766 (+ 2378us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 04:34:19.894210 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:19.887146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46017 (local address 127.0.97.62:43013)
0619 04:34:19.887297 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:19.887303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:19.887320 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:34:19.887377 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:19.887382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:19.887447 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:19.887541 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:19.888047 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.889019 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.890091 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.890629 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.890745 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:19.893995 (+ 3250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46017: BlockingRecv error: recv got EOF from 127.0.97.2:46017 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 04:34:19 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:19 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:19.901834 9498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:19.894669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:49115)
0619 04:34:19.894803 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:19.894820 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:19.894892 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:19.895177 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:19.895180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:19.895445 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:19.895585 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:19.895593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.896904 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.896908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:19.897777 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:19.897787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.898617 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.898620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:19.898730 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:19.899248 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:19.899263 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:19.901639 (+ 2376us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 04:34:19.902067 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:19.894771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49115 (local address 127.0.97.62:43013)
0619 04:34:19.894918 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:19.894924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:19.894941 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:34:19.894999 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:19.895003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:19.895070 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:19.895169 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:19.895723 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.896772 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.897905 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:19.898500 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:19.898616 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:19.901860 (+ 3244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49115: BlockingRecv error: recv got EOF from 127.0.97.2:49115 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 04:34:19.902221 9496 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 04:34:20.189822 9499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:20.190076 9499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:20.190191 9499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:20.197069 9499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:20.197161 9499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:20.197304 9499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:34:20.206564 9499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:38809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41949
--tserver_master_addrs=127.0.97.62:43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:20.207752 9499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:20.209002 9499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:20.221451 9504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:20.221478 9507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:20.221508 9505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:20.222333 9499 server_base.cc:1034] running on GCE node
I20260619 04:34:20.222846 9499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:20.223680 9499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:20.224862 9499 hybrid_clock.cc:648] HybridClock initialized: now 1781843660224822 us; error 26 us; skew 500 ppm
I20260619 04:34:20.225167 9499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:20 dist-test-slave-n3tq krb5kdc[8630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843660, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:20.228070 9499 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 04:34:20.229460 9499 webserver.cc:460] Webserver started at http://127.0.97.3:41949/ using document root <none> and password file <none>
I20260619 04:34:20.230116 9499 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:20.230237 9499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:20.235294 9499 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 04:34:20.237877 9514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:20.239166 9499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:34:20.239315 9499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "d37e49bfab544fa9bb4ecd68cc27c837"
format_stamp: "Formatted at 2026-06-19 04:34:17 on dist-test-slave-n3tq"
server_key: "07c72010832c3af83c21dd568dc7d5d8"
server_key_iv: "6a59f8e3620c069eda48034da0f1fc61"
server_key_version: "encryptionkey@0"
I20260619 04:34:20.240231 9499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:20.261193 9499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:20.265323 9499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:20.265578 9499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:20.266480 9499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:34:20.267741 9499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:20.267830 9499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:20.267938 9499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:20.268005 9499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:20.296937 9499 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38809
I20260619 04:34:20.296993 9626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38809 every 8 connection(s)
I20260619 04:34:20.298271 9499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:34:20.300686 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 9499
Jun 19 04:34:20 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:20 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 04:34:20.315474 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:20.300635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51107 (local address 127.0.97.62:43013)
0619 04:34:20.300840 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:20.300846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:20.302213 (+ 1367us) server_negotiation.cc:407] Connection header received
0619 04:34:20.305051 (+ 2838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:20.305057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:20.305153 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:20.305272 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:20.306909 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:20.307890 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:20.309675 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:20.310274 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:20.310366 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:20.315203 (+ 4837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51107: BlockingRecv error: recv got EOF from 127.0.97.3:51107 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 04:34:20.316108 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:20.301157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:51107)
0619 04:34:20.302022 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:20.302088 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:20.304700 (+ 2612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:20.305677 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:20.305687 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:20.306201 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:20.306744 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:20.306760 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:20.308023 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:20.308029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:20.309540 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:20.309551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:20.310494 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:20.310501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:20.310778 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:20.311446 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:20.311481 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:20.314982 (+ 3501us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":330,"threads_started":1}
W20260619 04:34:20.316716 9627 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43013 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:43013: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260619 04:34:20.319533 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:20.301624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34744 (local address 127.0.97.62:43013)
0619 04:34:20.305438 (+ 3814us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:20.305443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:20.305798 (+ 355us) server_negotiation.cc:407] Connection header received
0619 04:34:20.305959 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:20.305964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:20.306018 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:20.306111 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:20.307710 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:20.308771 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:20.310121 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:20.310655 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:20.310767 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:20.313526 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:20.313535 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:20.313551 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:20.313589 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:20.316402 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:20.317190 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:20.317197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:20.317203 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:20.317380 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:20.317754 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:20.317758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:20.317761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:20.318191 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:20.318506 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:20.318813 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 04:34:20.319168 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3709,"thread_start_us":3598,"threads_started":1}
Jun 19 04:34:20 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:20 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:20.324921 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:20.317229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:32995)
0619 04:34:20.317369 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:20.317386 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:20.317479 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:20.317745 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:20.317750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:20.318022 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:20.318146 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:20.318154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:20.319463 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:20.319467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:20.320615 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:20.320626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:20.321466 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:20.321470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:20.321608 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:20.322146 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:20.322161 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:20.324739 (+ 2578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 04:34:20.325075 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:20.317316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32995 (local address 127.0.97.62:43013)
0619 04:34:20.317465 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:20.317470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:20.317485 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:34:20.317582 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:20.317585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:20.317632 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:20.317696 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:20.318270 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:20.319319 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:20.320734 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:20.321343 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:20.321467 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:20.324884 (+ 3417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:32995: BlockingRecv error: recv got EOF from 127.0.97.3:32995 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 04:34:20 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:20 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:20.331869 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:20.325471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:44633)
0619 04:34:20.325608 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:20.325621 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:20.325690 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:20.325955 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:20.325958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:20.326184 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:20.326306 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:20.326313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:20.327248 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:20.327251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:20.328093 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:20.328102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:20.328710 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:20.328714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:20.328824 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:20.329290 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:20.329305 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:20.331604 (+ 2299us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 04:34:20.332136 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:20.325571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44633 (local address 127.0.97.62:43013)
0619 04:34:20.325758 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:20.325762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:20.325775 (+ 13us) server_negotiation.cc:407] Connection header received
0619 04:34:20.325808 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:20.325812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:20.325857 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:20.325926 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:20.326445 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:20.327148 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:20.328203 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:20.328616 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:20.328702 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:20.331791 (+ 3089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44633: BlockingRecv error: recv got EOF from 127.0.97.3:44633 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
W20260619 04:34:20.332119 9627 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 04:34:20 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:20 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:20.496307 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:20.488925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:55669)
0619 04:34:20.489349 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:20.489364 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:20.489473 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:20.489741 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:20.489743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:20.490159 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:20.490340 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:20.490348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:20.491267 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:20.491271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:20.492213 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:20.492224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:20.492903 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:20.492908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:20.493031 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:20.493602 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:20.493619 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:20.496090 (+ 2471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":220,"threads_started":1}
W20260619 04:34:20.496485 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:20.489020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55669 (local address 127.0.97.62:43013)
0619 04:34:20.489269 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:20.489275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:20.489434 (+ 159us) server_negotiation.cc:407] Connection header received
0619 04:34:20.489573 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:20.489577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:20.489635 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:20.489710 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:20.490447 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:20.491145 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:20.492332 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:20.492763 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:20.492850 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:20.496278 (+ 3428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55669: BlockingRecv error: recv got EOF from 127.0.97.1:55669 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 04:34:20 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:20 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:20.910913 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:20.902800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:59955)
0619 04:34:20.903258 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:20.903275 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:20.903385 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:20.903716 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:20.903720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:20.904242 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:20.904526 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:20.904543 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:20.905562 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:20.905566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:20.906496 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:20.906515 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:20.907125 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:20.907129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:20.907242 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:20.907971 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:20.907988 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:20.910615 (+ 2627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":240,"threads_started":1}
W20260619 04:34:20.911047 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:20.902869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59955 (local address 127.0.97.62:43013)
0619 04:34:20.903095 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:20.903099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:20.903358 (+ 259us) server_negotiation.cc:407] Connection header received
0619 04:34:20.903494 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:20.903497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:20.903558 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:20.903670 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:20.904658 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:20.905428 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:20.906622 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:20.907001 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:20.907086 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:20.910828 (+ 3742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59955: BlockingRecv error: recv got EOF from 127.0.97.2:59955 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 19 04:34:21 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:21 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:21.340523 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:21.332809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:59043)
0619 04:34:21.333306 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:21.333322 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:21.333434 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:21.333695 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:21.333698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:21.334122 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:21.334303 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:21.334312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:21.335280 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:21.335284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:21.336228 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:21.336240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:21.336913 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:21.336918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:21.337047 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:21.337672 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:21.337688 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:21.340289 (+ 2601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":268,"threads_started":1}
W20260619 04:34:21.340729 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:21.332870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59043 (local address 127.0.97.62:43013)
0619 04:34:21.333030 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:21.333035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:21.333409 (+ 374us) server_negotiation.cc:407] Connection header received
0619 04:34:21.333526 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:21.333529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:21.333602 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:21.333679 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:21.334441 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:21.335159 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:21.336334 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:21.336740 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:21.336823 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:21.340498 (+ 3675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59043: BlockingRecv error: recv got EOF from 127.0.97.3:59043 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 04:34:21 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:21 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:21.506012 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:21.497249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:51435)
0619 04:34:21.497773 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:21.497788 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:21.497902 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:21.498263 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:21.498266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:21.498622 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:21.498758 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:21.498766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:21.500145 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:21.500148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:21.501070 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:21.501081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:21.502006 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:21.502010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:21.502113 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:21.502739 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:21.502755 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:21.505768 (+ 3013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":275,"threads_started":1}
W20260619 04:34:21.506290 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:21.497346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51435 (local address 127.0.97.62:43013)
0619 04:34:21.497563 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:21.497571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:21.497861 (+ 290us) server_negotiation.cc:407] Connection header received
0619 04:34:21.498030 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:21.498036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:21.498130 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:21.498240 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:21.498876 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:21.500026 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:21.501193 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:21.501871 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:21.502009 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:21.506011 (+ 4002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51435: BlockingRecv error: recv got EOF from 127.0.97.1:51435 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 04:34:21 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:21 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:21.919809 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:21.911852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:59699)
0619 04:34:21.912482 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:21.912499 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:21.912632 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:21.912951 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:21.912957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:21.913384 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:21.913533 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:21.913542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:21.914515 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:21.914522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:21.915426 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:21.915438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:21.916132 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:21.916138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:21.916301 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:21.916889 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:21.916907 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:21.919474 (+ 2567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":352,"threads_started":1}
W20260619 04:34:21.919909 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:21.911917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59699 (local address 127.0.97.62:43013)
0619 04:34:21.912086 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:21.912092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:21.912580 (+ 488us) server_negotiation.cc:407] Connection header received
0619 04:34:21.912751 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:21.912756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:21.912824 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:21.912902 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:21.913673 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:21.914389 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:21.915563 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:21.915988 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:21.916091 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:21.919707 (+ 3616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59699: BlockingRecv error: recv got EOF from 127.0.97.2:59699 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 04:34:22 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:22 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:22.349121 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:22.341347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:44957)
0619 04:34:22.341833 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:22.341848 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:22.341976 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:22.342280 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:22.342285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:22.342685 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:22.342823 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:22.342831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:22.343849 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:22.343855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:22.344816 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:22.344828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:22.345458 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:22.345464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:22.345608 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:22.346191 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:22.346207 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:22.348879 (+ 2672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":259,"threads_started":1}
W20260619 04:34:22.349363 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:22.341540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44957 (local address 127.0.97.62:43013)
0619 04:34:22.341760 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:22.341766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:22.341927 (+ 161us) server_negotiation.cc:407] Connection header received
0619 04:34:22.342088 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:22.342093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:22.342157 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:22.342235 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:22.342982 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:22.343722 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:22.344950 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:22.345323 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:22.345425 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:22.349113 (+ 3688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44957: BlockingRecv error: recv got EOF from 127.0.97.3:44957 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 19 04:34:22 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:22 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:22.514237 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:22.506902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:53855)
0619 04:34:22.507376 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:22.507391 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:22.507508 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:22.507792 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:22.507796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:22.508128 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:22.508289 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:22.508297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:22.509252 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:22.509255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:22.510140 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:22.510152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:22.510744 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:22.510747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:22.510861 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:22.511484 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:22.511501 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:22.513979 (+ 2478us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":268,"threads_started":1}
W20260619 04:34:22.514385 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:22.506979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53855 (local address 127.0.97.62:43013)
0619 04:34:22.507143 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:22.507147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:22.507452 (+ 305us) server_negotiation.cc:407] Connection header received
0619 04:34:22.507592 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:22.507595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:22.507661 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:22.507739 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:22.508398 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:22.509137 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:22.510255 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:22.510642 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:22.510727 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:22.514145 (+ 3418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53855: BlockingRecv error: recv got EOF from 127.0.97.1:53855 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 04:34:22 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:22 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:22.929035 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:22.920843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:46577)
0619 04:34:22.921329 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:22.921345 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:22.921476 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:22.921808 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:22.921813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:22.922160 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:22.922324 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:22.922333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:22.923335 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:22.923340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:22.924237 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:22.924248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:22.924953 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:22.924958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:22.925124 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:22.925703 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:22.925721 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:22.928709 (+ 2988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":284,"threads_started":1}
W20260619 04:34:22.929147 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:22.920916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46577 (local address 127.0.97.62:43013)
0619 04:34:22.921163 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:22.921169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:22.921433 (+ 264us) server_negotiation.cc:407] Connection header received
0619 04:34:22.921596 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:22.921602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:22.921676 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:22.921757 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:22.922468 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:22.923210 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:22.924373 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:22.924815 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:22.924923 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:22.928925 (+ 4002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46577: BlockingRecv error: recv got EOF from 127.0.97.2:46577 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 04:34:23 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:23 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:23.358032 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:23.350092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:55353)
0619 04:34:23.350547 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:23.350567 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:23.350699 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:23.351081 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:23.351087 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:23.351538 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:23.351742 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:23.351754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:23.352918 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:23.352922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:23.353824 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:23.353836 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:23.354445 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:23.354450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:23.354579 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:23.355088 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:23.355103 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:23.357708 (+ 2605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":247,"threads_started":1}
W20260619 04:34:23.358122 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:23.350255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55353 (local address 127.0.97.62:43013)
0619 04:34:23.350439 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:23.350446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:23.350655 (+ 209us) server_negotiation.cc:407] Connection header received
0619 04:34:23.350828 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:23.350832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:23.350907 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:23.351002 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:23.351900 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:23.352800 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:23.353938 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:23.354321 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:23.354409 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:23.357919 (+ 3510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55353: BlockingRecv error: recv got EOF from 127.0.97.3:55353 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 04:34:23 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:23 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:23.522940 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:23.515139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:44085)
0619 04:34:23.515703 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:23.515719 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:23.515823 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:23.516076 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:23.516079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:23.516405 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:23.516577 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:23.516586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:23.517538 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:23.517542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:23.518490 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:23.518502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:23.519158 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:23.519162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:23.519289 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:23.519920 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:23.519936 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:23.522719 (+ 2783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":345,"threads_started":1}
W20260619 04:34:23.523182 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:23.515172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44085 (local address 127.0.97.62:43013)
0619 04:34:23.515362 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:23.515367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:23.515792 (+ 425us) server_negotiation.cc:407] Connection header received
0619 04:34:23.515915 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:23.515919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:23.515982 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:23.516056 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:23.516698 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:23.517408 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:23.518635 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:23.519045 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:23.519145 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:23.522934 (+ 3789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44085: BlockingRecv error: recv got EOF from 127.0.97.1:44085 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 04:34:23 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:23 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:23.937734 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:23.930084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:36007)
0619 04:34:23.930573 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:23.930589 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:23.930699 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:23.931007 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:23.931011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:23.931372 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:23.931521 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:23.931529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:23.932413 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:23.932417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:23.933403 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:23.933414 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:23.934056 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:23.934060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:23.934209 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:23.934823 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:23.934840 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:23.937504 (+ 2664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":282,"threads_started":1}
W20260619 04:34:23.937883 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:23.930154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36007 (local address 127.0.97.62:43013)
0619 04:34:23.930387 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:23.930393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:23.930670 (+ 277us) server_negotiation.cc:407] Connection header received
0619 04:34:23.930795 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:23.930799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:23.930859 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:23.930930 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:23.931629 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:23.932297 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:23.933512 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:23.933919 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:23.934005 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:23.937699 (+ 3694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36007: BlockingRecv error: recv got EOF from 127.0.97.2:36007 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 04:34:24 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:24 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:24.367111 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:24.358979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:50593)
0619 04:34:24.359502 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:24.359517 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:24.359639 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:24.359942 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:24.359946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:24.360379 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:24.360620 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:24.360631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:24.361853 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:24.361859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:24.362806 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:24.362817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:24.363438 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:24.363444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:24.363621 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:24.364223 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:24.364240 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:24.366812 (+ 2572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":312,"threads_started":1}
W20260619 04:34:24.367257 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:24.359099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50593 (local address 127.0.97.62:43013)
0619 04:34:24.359277 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:24.359282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:24.359621 (+ 339us) server_negotiation.cc:407] Connection header received
0619 04:34:24.359734 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:24.359738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:24.359801 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:24.359884 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:24.360748 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:24.361730 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:24.362932 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:24.363310 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:24.363409 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:24.367014 (+ 3605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50593: BlockingRecv error: recv got EOF from 127.0.97.3:50593 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 04:34:24 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:24 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:24.531284 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:24.523819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:34103)
0619 04:34:24.524285 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:24.524299 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:24.524418 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:24.524764 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:24.524770 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:24.525163 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:24.525312 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:24.525320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:24.526283 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:24.526288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:24.527173 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:24.527184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:24.527863 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:24.527868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:24.528013 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:24.528541 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:24.528557 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:24.530955 (+ 2398us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":223,"threads_started":1}
W20260619 04:34:24.531351 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:24.523910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34103 (local address 127.0.97.62:43013)
0619 04:34:24.524120 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:24.524126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:24.524376 (+ 250us) server_negotiation.cc:407] Connection header received
0619 04:34:24.524552 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:24.524557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:24.524629 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:24.524713 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:24.525445 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:24.526151 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:24.527296 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:24.527737 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:24.527839 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:24.531154 (+ 3315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:34103: BlockingRecv error: recv got EOF from 127.0.97.1:34103 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 04:34:24 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:24 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:24.950977 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:24.938680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:46423)
0619 04:34:24.939201 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:24.939218 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:24.939332 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:24.939592 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:24.939595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:24.939959 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:24.940110 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:24.940118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:24.941482 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:24.941488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:24.942574 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:24.942587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:24.943297 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:24.943302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:24.943418 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:24.947919 (+ 4501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:24.947938 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:24.950650 (+ 2712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":265,"threads_started":1}
W20260619 04:34:24.951140 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:24.938886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46423 (local address 127.0.97.62:43013)
0619 04:34:24.939080 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:24.939085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:24.939293 (+ 208us) server_negotiation.cc:407] Connection header received
0619 04:34:24.939423 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:24.939427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:24.939483 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:24.939558 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:24.940278 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:24.941312 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:24.942695 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:24.943166 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:24.943258 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:24.950890 (+ 7632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46423: BlockingRecv error: recv got EOF from 127.0.97.2:46423 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 04:34:25 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:25 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:25.385586 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:25.368048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:46851)
0619 04:34:25.368544 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:25.368561 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:25.368683 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:25.368976 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:25.368980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:25.369355 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:25.369503 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:25.369512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:25.370483 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:25.370488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:25.371371 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:25.371381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:25.372009 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:25.372014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:25.372178 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:25.382592 (+ 10414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:25.382614 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:25.385264 (+ 2650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":228,"threads_started":1}
W20260619 04:34:25.385692 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:25.368108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46851 (local address 127.0.97.62:43013)
0619 04:34:25.368280 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:25.368286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:25.368641 (+ 355us) server_negotiation.cc:407] Connection header received
0619 04:34:25.368796 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:25.368800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:25.368856 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:25.368932 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:25.369647 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:25.370349 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:25.371500 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:25.371882 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:25.371983 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:25.385474 (+ 13491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46851: BlockingRecv error: recv got EOF from 127.0.97.3:46851 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 04:34:25 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:25 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:25.539458 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:25.532073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:59351)
0619 04:34:25.532505 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:25.532521 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:25.532629 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:25.532911 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:25.532914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:25.533262 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:25.533403 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:25.533411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:25.534375 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:25.534381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:25.535269 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:25.535281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:25.535891 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:25.535895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:25.536008 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:25.536619 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:25.536635 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:25.539161 (+ 2526us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":235,"threads_started":1}
W20260619 04:34:25.539592 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:25.532145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59351 (local address 127.0.97.62:43013)
0619 04:34:25.532336 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:25.532342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:25.532595 (+ 253us) server_negotiation.cc:407] Connection header received
0619 04:34:25.532745 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:25.532748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:25.532808 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:25.532881 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:25.533501 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:25.534232 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:25.535380 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:25.535781 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:25.535863 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:25.539355 (+ 3492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59351: BlockingRecv error: recv got EOF from 127.0.97.1:59351 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 04:34:25 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:25 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:25.959371 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:25.951837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:50291)
0619 04:34:25.952561 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:25.952579 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:25.952691 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:25.952976 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:25.952979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:25.953343 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:25.953482 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:25.953489 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:25.954506 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:25.954510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:25.955420 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:25.955431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:25.956041 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:25.956045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:25.956153 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:25.956722 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:25.956739 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:25.959155 (+ 2416us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":491,"threads_started":1}
W20260619 04:34:25.959609 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:25.951885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50291 (local address 127.0.97.62:43013)
0619 04:34:25.952053 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:25.952058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:25.952651 (+ 593us) server_negotiation.cc:407] Connection header received
0619 04:34:25.952809 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:25.952813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:25.952872 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:25.952946 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:25.953652 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:25.954373 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:25.955530 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:25.955926 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:25.956012 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:25.959370 (+ 3358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50291: BlockingRecv error: recv got EOF from 127.0.97.2:50291 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 04:34:26 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:26 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:26.394325 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:26.386577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:39435)
0619 04:34:26.387022 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:26.387045 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:26.387169 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:26.387465 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:26.387468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:26.387818 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:26.387961 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:26.387969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:26.389136 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:26.389141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:26.389994 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:26.390005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:26.390701 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:26.390706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:26.390856 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:26.391425 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:26.391441 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:26.394109 (+ 2668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":253,"threads_started":1}
W20260619 04:34:26.394512 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:26.386692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39435 (local address 127.0.97.62:43013)
0619 04:34:26.386916 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:26.386923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:26.387148 (+ 225us) server_negotiation.cc:407] Connection header received
0619 04:34:26.387275 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:26.387279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:26.387349 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:26.387437 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:26.388107 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:26.388999 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:26.390143 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:26.390562 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:26.390665 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:26.394310 (+ 3645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39435: BlockingRecv error: recv got EOF from 127.0.97.3:39435 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 19 04:34:26 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:26 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:26.548441 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:26.540429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:50055)
0619 04:34:26.541054 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:26.541073 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:26.541217 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:26.541557 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:26.541561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:26.541975 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:26.542146 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:26.542155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:26.543128 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:26.543132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:26.544062 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:26.544073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:26.544768 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:26.544773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:26.544889 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:26.545467 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:26.545490 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:26.548214 (+ 2724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":254,"threads_started":1}
W20260619 04:34:26.548696 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:26.540566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50055 (local address 127.0.97.62:43013)
0619 04:34:26.540786 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:26.540793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:26.541169 (+ 376us) server_negotiation.cc:407] Connection header received
0619 04:34:26.541329 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:26.541334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:26.541410 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:26.541493 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:26.542301 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:26.542995 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:26.544181 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:26.544623 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:26.544748 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:26.548434 (+ 3686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50055: BlockingRecv error: recv got EOF from 127.0.97.1:50055 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 04:34:26 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:26 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:26.967849 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:26.960322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:43645)
0619 04:34:26.960832 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:26.960847 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:26.960954 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:26.961270 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:26.961273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:26.961669 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:26.961834 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:26.961843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:26.962804 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:26.962807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:26.963689 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:26.963700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:26.964308 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:26.964312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:26.964414 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:26.965012 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:26.965030 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:26.967607 (+ 2577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":221,"threads_started":1}
W20260619 04:34:26.967975 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:26.960347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43645 (local address 127.0.97.62:43013)
0619 04:34:26.960618 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:26.960625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:26.960903 (+ 278us) server_negotiation.cc:407] Connection header received
0619 04:34:26.961049 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:26.961053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:26.961113 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:26.961224 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:26.961956 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:26.962696 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:26.963790 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:26.964200 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:26.964282 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:26.967775 (+ 3493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43645: BlockingRecv error: recv got EOF from 127.0.97.2:43645 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Jun 19 04:34:27 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:27 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:27.403000 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:27.395159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:57693)
0619 04:34:27.395710 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:27.395738 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:27.395853 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:27.396108 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:27.396112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:27.396526 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:27.396692 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:27.396700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:27.397746 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:27.397751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:27.398701 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:27.398715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:27.399344 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:27.399349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:27.399455 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:27.400036 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:27.400054 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:27.402758 (+ 2704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":233,"threads_started":1}
W20260619 04:34:27.403147 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:27.395242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57693 (local address 127.0.97.62:43013)
0619 04:34:27.395412 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:27.395418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:27.395801 (+ 383us) server_negotiation.cc:407] Connection header received
0619 04:34:27.395943 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:27.395946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:27.396006 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:27.396083 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:27.396846 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:27.397597 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:27.398815 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:27.399223 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:27.399323 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:27.402955 (+ 3632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57693: BlockingRecv error: recv got EOF from 127.0.97.3:57693 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 04:34:27 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:27 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:27.556598 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:27.549372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:38207)
0619 04:34:27.549815 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:27.549831 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:27.549937 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:27.550189 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:27.550192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:27.550555 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:27.550702 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:27.550709 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:27.551709 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:27.551713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:27.552680 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:27.552692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:27.553302 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:27.553306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:27.553414 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:27.553936 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:27.553952 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:27.556355 (+ 2403us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":236,"threads_started":1}
W20260619 04:34:27.556777 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:27.549506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38207 (local address 127.0.97.62:43013)
0619 04:34:27.549654 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:27.549658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:27.549890 (+ 232us) server_negotiation.cc:407] Connection header received
0619 04:34:27.550028 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:27.550032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:27.550088 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:27.550164 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:27.550823 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:27.551587 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:27.552782 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:27.553158 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:27.553246 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:27.556556 (+ 3310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38207: BlockingRecv error: recv got EOF from 127.0.97.1:38207 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 04:34:27 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:27 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:27.976594 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:27.968687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:57929)
0619 04:34:27.969177 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:27.969197 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:27.969325 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:27.969623 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:27.969629 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:27.970062 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:27.970210 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:27.970220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:27.971469 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:27.971475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:27.972375 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:27.972386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:27.973074 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:27.973079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:27.973231 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:27.973784 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:27.973801 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:27.976276 (+ 2475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":251,"threads_started":1}
W20260619 04:34:27.976691 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:27.968773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57929 (local address 127.0.97.62:43013)
0619 04:34:27.968950 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:27.968955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:27.969283 (+ 328us) server_negotiation.cc:407] Connection header received
0619 04:34:27.969442 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:27.969445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:27.969503 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:27.969582 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:27.970343 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:27.971337 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:27.972505 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:27.972913 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:27.973016 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:27.976499 (+ 3483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57929: BlockingRecv error: recv got EOF from 127.0.97.2:57929 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 04:34:28 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:28 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:28.411757 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:28.403838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:56345)
0619 04:34:28.404440 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:28.404477 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:28.404611 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:28.404911 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:28.404916 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:28.405342 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:28.405490 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:28.405500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:28.406484 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:28.406489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:28.407440 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:28.407452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:28.408198 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:28.408203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:28.408324 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:28.408959 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:28.408976 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:28.411528 (+ 2552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":394,"threads_started":1}
W20260619 04:34:28.411942 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:28.403932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56345 (local address 127.0.97.62:43013)
0619 04:34:28.404177 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:28.404184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:28.404547 (+ 363us) server_negotiation.cc:407] Connection header received
0619 04:34:28.404715 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:28.404720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:28.404790 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:28.404878 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:28.405623 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:28.406335 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:28.407599 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:28.408013 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:28.408130 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:28.411725 (+ 3595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56345: BlockingRecv error: recv got EOF from 127.0.97.3:56345 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 19 04:34:28 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:28 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:28.566025 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:28.557447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:39027)
0619 04:34:28.557916 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:28.557932 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:28.558055 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:28.558441 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:28.558446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:28.558877 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:28.559033 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:28.559041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:28.560014 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:28.560020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:28.561048 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:28.561063 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:28.562036 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:28.562044 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:28.562230 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:28.562869 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:28.562891 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:28.565695 (+ 2804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":248,"threads_started":1}
W20260619 04:34:28.566102 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:28.557574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39027 (local address 127.0.97.62:43013)
0619 04:34:28.557757 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:28.557764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:28.557996 (+ 232us) server_negotiation.cc:407] Connection header received
0619 04:34:28.558198 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:28.558206 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:28.558302 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:28.558399 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:28.559173 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:28.559890 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:28.561173 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:28.561844 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:28.561958 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:28.565901 (+ 3943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39027: BlockingRecv error: recv got EOF from 127.0.97.1:39027 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 04:34:28 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:28 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:28.985534 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:28.977467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:37957)
0619 04:34:28.977916 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:28.977931 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:28.978047 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:28.978297 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:28.978300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:28.978644 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:28.978806 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:28.978815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:28.979717 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:28.979720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:28.980690 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:28.980701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:28.981467 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:28.981471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:28.981595 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:28.982222 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:28.982244 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:28.985292 (+ 3048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":236,"threads_started":1}
W20260619 04:34:28.985670 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:28.977569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37957 (local address 127.0.97.62:43013)
0619 04:34:28.977791 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:28.977797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:28.977998 (+ 201us) server_negotiation.cc:407] Connection header received
0619 04:34:28.978125 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:28.978129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:28.978190 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:28.978266 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:28.978923 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:28.979619 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:28.980821 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:28.981330 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:28.981442 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:28.985468 (+ 4026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37957: BlockingRecv error: recv got EOF from 127.0.97.2:37957 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 19 04:34:29 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:29 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:29.420681 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:29.412632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:37827)
0619 04:34:29.413131 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:29.413146 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:29.413260 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:29.413575 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:29.413579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:29.413937 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:29.414094 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:29.414103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:29.415040 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:29.415044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:29.416031 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:29.416043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:29.416843 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:29.416848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:29.417015 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:29.417613 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:29.417629 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:29.420319 (+ 2690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":245,"threads_started":1}
W20260619 04:34:29.420717 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:29.412687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37827 (local address 127.0.97.62:43013)
0619 04:34:29.412841 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:29.412845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:29.413222 (+ 377us) server_negotiation.cc:407] Connection header received
0619 04:34:29.413349 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:29.413353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:29.413414 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:29.413493 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:29.414220 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:29.414918 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:29.416159 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:29.416698 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:29.416802 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:29.420549 (+ 3747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37827: BlockingRecv error: recv got EOF from 127.0.97.3:37827 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 04:34:29 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:29 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:29.574952 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:29.566965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:48623)
0619 04:34:29.567446 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:29.567461 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:29.567568 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:29.567880 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:29.567884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:29.568259 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:29.568400 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:29.568409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:29.569476 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:29.569480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:29.570437 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:29.570449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:29.571157 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:29.571162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:29.571347 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:29.571968 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:29.571984 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:29.574710 (+ 2726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":280,"threads_started":1}
W20260619 04:34:29.575224 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:29.567035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48623 (local address 127.0.97.62:43013)
0619 04:34:29.567186 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:29.567191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:29.567556 (+ 365us) server_negotiation.cc:407] Connection header received
0619 04:34:29.567667 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:29.567670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:29.567748 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:29.567830 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:29.568561 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:29.569309 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:29.570543 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:29.571027 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:29.571147 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:29.574915 (+ 3768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48623: BlockingRecv error: recv got EOF from 127.0.97.1:48623 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 04:34:29 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:29 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:29.996402 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:29.986405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:53685)
0619 04:34:29.986852 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:29.986871 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:29.986986 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:29.987303 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:29.987308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:29.987642 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:29.987781 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:29.987789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:29.988811 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:29.988816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:29.989761 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:29.989773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:29.990387 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:29.990391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:29.990509 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:29.993429 (+ 2920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:29.993448 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:29.996104 (+ 2656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":245,"threads_started":1}
W20260619 04:34:29.996608 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:29.986482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53685 (local address 127.0.97.62:43013)
0619 04:34:29.986630 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:29.986635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:29.986945 (+ 310us) server_negotiation.cc:407] Connection header received
0619 04:34:29.987085 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:29.987089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:29.987157 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:29.987249 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:29.987907 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:29.988657 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:29.989871 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:29.990276 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:29.990360 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:29.996333 (+ 5973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53685: BlockingRecv error: recv got EOF from 127.0.97.2:53685 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 04:34:30 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:30 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:30.429011 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:30.421544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:56325)
0619 04:34:30.422021 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:30.422038 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:30.422143 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:30.422428 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:30.422432 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:30.422793 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:30.422930 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:30.422938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:30.423856 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:30.423860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:30.424804 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:30.424814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:30.425540 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:30.425545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:30.425668 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:30.426255 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:30.426291 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:30.428800 (+ 2509us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":269,"threads_started":1}
W20260619 04:34:30.429265 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:30.421699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56325 (local address 127.0.97.62:43013)
0619 04:34:30.421855 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:30.421859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:30.422098 (+ 239us) server_negotiation.cc:407] Connection header received
0619 04:34:30.422244 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:30.422247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:30.422321 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:30.422395 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:30.423037 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:30.423738 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:30.424932 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:30.425348 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:30.425439 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:30.429018 (+ 3579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56325: BlockingRecv error: recv got EOF from 127.0.97.3:56325 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 04:34:30 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:30 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:30.583640 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:30.575859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:59447)
0619 04:34:30.576302 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:30.576318 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:30.576474 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:30.576755 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:30.576758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:30.577125 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:30.577301 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:30.577310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:30.578284 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:30.578289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:30.579199 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:30.579214 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:30.579908 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:30.579913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:30.580044 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:30.580686 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:30.580704 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:30.583390 (+ 2686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":231,"threads_started":1}
W20260619 04:34:30.583791 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:30.575927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59447 (local address 127.0.97.62:43013)
0619 04:34:30.576093 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:30.576098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:30.576399 (+ 301us) server_negotiation.cc:407] Connection header received
0619 04:34:30.576571 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:30.576575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:30.576643 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:30.576724 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:30.577408 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:30.578145 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:30.579332 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:30.579736 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:30.579830 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:30.583592 (+ 3762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59447: BlockingRecv error: recv got EOF from 127.0.97.1:59447 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 04:34:31 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:31 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:31.006314 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:30.997428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:45163)
0619 04:34:30.997883 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:30.997900 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:30.998017 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:30.998375 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:30.998379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:30.998765 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:30.998942 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:30.998951 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:31.000448 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:31.000470 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:31.001707 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:31.001721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:31.002692 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:31.002697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:31.002832 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:31.003455 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:31.003472 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:31.006063 (+ 2591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":231,"threads_started":1}
W20260619 04:34:31.006480 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:30.997535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45163 (local address 127.0.97.62:43013)
0619 04:34:30.997729 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:30.997735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:30.997974 (+ 239us) server_negotiation.cc:407] Connection header received
0619 04:34:30.998113 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:30.998119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:30.998193 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:30.998299 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:30.999078 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:31.000274 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:31.001888 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:31.002535 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:31.002649 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:31.006269 (+ 3620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45163: BlockingRecv error: recv got EOF from 127.0.97.2:45163 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 04:34:31 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:31 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:31.437708 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:31.429933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:46323)
0619 04:34:31.430525 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:31.430540 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:31.430651 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:31.430990 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:31.430996 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:31.431426 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:31.431593 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:31.431602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:31.432568 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:31.432572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:31.433463 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:31.433475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:31.434069 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:31.434073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:31.434180 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:31.434720 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:31.434735 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:31.437387 (+ 2652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":377,"threads_started":1}
W20260619 04:34:31.437795 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:31.429990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46323 (local address 127.0.97.62:43013)
0619 04:34:31.430166 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:31.430172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:31.430618 (+ 446us) server_negotiation.cc:407] Connection header received
0619 04:34:31.430778 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:31.430781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:31.430849 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:31.430950 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:31.431705 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:31.432423 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:31.433575 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:31.433963 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:31.434048 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:31.437579 (+ 3531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46323: BlockingRecv error: recv got EOF from 127.0.97.3:46323 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 04:34:31 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:31 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:31.592387 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:31.584698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:33727)
0619 04:34:31.585143 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:31.585158 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:31.585278 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:31.585559 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:31.585562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:31.585956 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:31.586137 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:31.586147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:31.587348 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:31.587354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:31.588249 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:31.588260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:31.588951 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:31.588954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:31.589075 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:31.589626 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:31.589643 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:31.592164 (+ 2521us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":232,"threads_started":1}
W20260619 04:34:31.592643 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:31.584759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33727 (local address 127.0.97.62:43013)
0619 04:34:31.584972 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:31.584978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:31.585236 (+ 258us) server_negotiation.cc:407] Connection header received
0619 04:34:31.585370 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:31.585373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:31.585445 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:31.585524 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:31.586266 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:31.587217 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:31.588357 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:31.588848 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:31.588938 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:31.592381 (+ 3443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33727: BlockingRecv error: recv got EOF from 127.0.97.1:33727 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 04:34:32 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:32 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:32.014359 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:32.007118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:51409)
0619 04:34:32.007560 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:32.007577 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:32.007688 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:32.007957 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:32.007961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:32.008305 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:32.008481 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:32.008489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:32.009381 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:32.009385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:32.010284 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:32.010296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:32.010977 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:32.010981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:32.011110 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:32.011657 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:32.011673 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:32.014130 (+ 2457us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":229,"threads_started":1}
W20260619 04:34:32.014472 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:32.007162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51409 (local address 127.0.97.62:43013)
0619 04:34:32.007358 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:32.007365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:32.007654 (+ 289us) server_negotiation.cc:407] Connection header received
0619 04:34:32.007787 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:32.007791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:32.007850 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:32.007924 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:32.008582 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:32.009255 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:32.010426 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:32.010847 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:32.010949 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:32.014307 (+ 3358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51409: BlockingRecv error: recv got EOF from 127.0.97.2:51409 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 04:34:32 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:32 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:32.446491 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:32.438571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:56763)
0619 04:34:32.439060 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:32.439075 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:32.439215 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:32.439485 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:32.439488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:32.439847 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:32.439994 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:32.440003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:32.441026 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:32.441030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:32.441969 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:32.441981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:32.442572 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:32.442576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:32.442694 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:32.443244 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:32.443262 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:32.446270 (+ 3008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":257,"threads_started":1}
W20260619 04:34:32.446664 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:32.438645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56763 (local address 127.0.97.62:43013)
0619 04:34:32.438796 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:32.438801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:32.439137 (+ 336us) server_negotiation.cc:407] Connection header received
0619 04:34:32.439304 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:32.439307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:32.439366 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:32.439453 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:32.440102 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:32.440885 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:32.442080 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:32.442471 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:32.442554 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:32.446462 (+ 3908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56763: BlockingRecv error: recv got EOF from 127.0.97.3:56763 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 04:34:32 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:32 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:32.602118 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:32.593265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:48015)
0619 04:34:32.594035 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:32.594051 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:32.594269 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:32.594796 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:32.594801 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:32.595447 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:32.595603 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:32.595612 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:32.596748 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:32.596752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:32.597705 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:32.597717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:32.598381 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:32.598385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:32.598494 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:32.599020 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:32.599035 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:32.601825 (+ 2790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":383,"threads_started":1}
W20260619 04:34:32.602283 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:32.593400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48015 (local address 127.0.97.62:43013)
0619 04:34:32.593618 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:32.593626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:32.594168 (+ 542us) server_negotiation.cc:407] Connection header received
0619 04:34:32.594484 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:32.594489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:32.594580 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:32.594667 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:32.595807 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:32.596611 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:32.597827 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:32.598230 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:32.598319 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:32.602035 (+ 3716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48015: BlockingRecv error: recv got EOF from 127.0.97.1:48015 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 04:34:33 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:33 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:33.023722 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:33.015352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:58667)
0619 04:34:33.015864 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:33.015879 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:33.016010 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:33.016361 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:33.016366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:33.016777 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:33.016968 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:33.016979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:33.018191 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:33.018197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:33.019270 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:33.019283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:33.020053 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:33.020058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:33.020189 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:33.020789 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:33.020806 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:33.023420 (+ 2614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":246,"threads_started":1}
W20260619 04:34:33.023840 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:33.015528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58667 (local address 127.0.97.62:43013)
0619 04:34:33.015747 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:33.015754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:33.015958 (+ 204us) server_negotiation.cc:407] Connection header received
0619 04:34:33.016125 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:33.016129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:33.016228 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:33.016308 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:33.017130 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:33.018044 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:33.019411 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:33.019885 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:33.019994 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:33.023623 (+ 3629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58667: BlockingRecv error: recv got EOF from 127.0.97.2:58667 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 19 04:34:33 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:33 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:33.455403 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:33.447402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:60383)
0619 04:34:33.447855 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:33.447871 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:33.447997 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:33.448313 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:33.448318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:33.448730 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:33.448895 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:33.448905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:33.449878 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:33.449881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:33.450801 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:33.450812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:33.451509 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:33.451514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:33.451663 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:33.452258 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:33.452275 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:33.455179 (+ 2904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":223,"threads_started":1}
W20260619 04:34:33.455687 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:33.447411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60383 (local address 127.0.97.62:43013)
0619 04:34:33.447583 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:33.447589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:33.447955 (+ 366us) server_negotiation.cc:407] Connection header received
0619 04:34:33.448107 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:33.448112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:33.448177 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:33.448276 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:33.449030 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:33.449737 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:33.450933 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:33.451346 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:33.451450 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:33.455473 (+ 4023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60383: BlockingRecv error: recv got EOF from 127.0.97.3:60383 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 04:34:33 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:33 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:33.611169 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:33.603216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:52849)
0619 04:34:33.603836 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:33.603851 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:33.603976 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:33.604308 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:33.604314 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:33.604715 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:33.604930 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:33.604938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:33.605912 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:33.605917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:33.606857 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:33.606870 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:33.607467 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:33.607471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:33.607583 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:33.608170 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:33.608186 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:33.610859 (+ 2673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":377,"threads_started":1}
W20260619 04:34:33.611320 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:33.603256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52849 (local address 127.0.97.62:43013)
0619 04:34:33.603539 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:33.603546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:33.603926 (+ 380us) server_negotiation.cc:407] Connection header received
0619 04:34:33.604081 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:33.604087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:33.604171 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:33.604265 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:33.605044 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:33.605758 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:33.606974 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:33.607360 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:33.607448 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:33.611079 (+ 3631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52849: BlockingRecv error: recv got EOF from 127.0.97.1:52849 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
Jun 19 04:34:34 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:34 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:34.032578 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:34.024736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:60603)
0619 04:34:34.025200 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:34.025217 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:34.025328 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:34.025636 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:34.025640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:34.026086 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:34.026256 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:34.026265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:34.027267 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:34.027272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:34.028227 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:34.028239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:34.028988 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:34.028993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:34.029156 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:34.029698 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:34.029715 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:34.032295 (+ 2580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":252,"threads_started":1}
W20260619 04:34:34.032692 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:34.024823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60603 (local address 127.0.97.62:43013)
0619 04:34:34.024992 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:34.024997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:34.025324 (+ 327us) server_negotiation.cc:407] Connection header received
0619 04:34:34.025424 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:34.025430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:34.025494 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:34.025580 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:34.026395 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:34.027132 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:34.028362 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:34.028856 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:34.028962 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:34.032518 (+ 3556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60603: BlockingRecv error: recv got EOF from 127.0.97.2:60603 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 04:34:34 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:34 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:34.464222 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:34.456369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:40273)
0619 04:34:34.456853 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:34.456869 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:34.456996 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:34.457331 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:34.457337 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:34.457764 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:34.457985 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:34.457996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:34.459017 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:34.459021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:34.459972 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:34.459984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:34.460762 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:34.460766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:34.460918 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:34.461472 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:34.461489 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:34.463984 (+ 2495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":272,"threads_started":1}
W20260619 04:34:34.464390 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:34.456528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40273 (local address 127.0.97.62:43013)
0619 04:34:34.456721 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:34.456727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:34.456960 (+ 233us) server_negotiation.cc:407] Connection header received
0619 04:34:34.457112 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:34.457116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:34.457179 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:34.457276 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:34.458110 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:34.458884 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:34.460118 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:34.460612 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:34.460717 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:34.464197 (+ 3480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40273: BlockingRecv error: recv got EOF from 127.0.97.3:40273 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 04:34:34 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:34 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:34.620316 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:34.612395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:56573)
0619 04:34:34.612889 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:34.612907 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:34.613042 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:34.613422 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:34.613427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:34.613947 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:34.614164 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:34.614177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:34.615356 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:34.615360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:34.616258 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:34.616270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:34.616938 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:34.616942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:34.617057 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:34.617575 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:34.617591 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:34.620037 (+ 2446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":315,"threads_started":1}
W20260619 04:34:34.620651 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:34.612490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56573 (local address 127.0.97.62:43013)
0619 04:34:34.612702 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:34.612709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:34.612989 (+ 280us) server_negotiation.cc:407] Connection header received
0619 04:34:34.613186 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:34.613191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:34.613261 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:34.613356 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:34.614364 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:34.615223 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:34.616362 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:34.616814 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:34.616898 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:34.620221 (+ 3323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56573: BlockingRecv error: recv got EOF from 127.0.97.1:56573 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 19 04:34:35 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:34:35 dist-test-slave-n3tq krb5kdc[8630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:34:35.044564 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:35.033578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:57301)
0619 04:34:35.034042 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:35.034069 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:35.034190 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:35.034505 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:35.034510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:35.034904 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:35.035055 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:35.035064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:35.036016 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:35.036020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:35.036969 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:35.036980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:35.037690 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:35.037695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:35.037847 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:35.041497 (+ 3650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:35.041513 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:35.044222 (+ 2709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43013: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":255,"threads_started":1}
W20260619 04:34:35.044701 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:35.033648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57301 (local address 127.0.97.62:43013)
0619 04:34:35.033837 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:35.033843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:35.034147 (+ 304us) server_negotiation.cc:407] Connection header received
0619 04:34:35.034301 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:35.034306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:35.034374 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:35.034464 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:35.035179 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:35.035883 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:35.037095 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:35.037561 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:35.037666 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:35.044433 (+ 6767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57301: BlockingRecv error: recv got EOF from 127.0.97.2:57301 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 04:34:35.305024 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 9217
I20260619 04:34:35.316246 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:35.306124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49722 (local address 127.0.97.1:34419)
0619 04:34:35.306689 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:35.306696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:35.306723 (+ 27us) server_negotiation.cc:407] Connection header received
0619 04:34:35.306780 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:35.306786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:35.307007 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:35.307160 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:35.308483 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:35.309545 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:35.310681 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:35.311232 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:35.311356 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:35.313538 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:35.313546 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:35.313561 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:35.313594 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:35.315771 (+ 2177us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 04:34:35.316016 (+ 245us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49722: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":293,"threads_started":1}
W20260619 04:34:35.316354 9677 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49722: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 04:34:35.316393 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 9217: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:34419: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 04:34:35.316588 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 9217
I20260619 04:34:35.330037 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 9368
I20260619 04:34:35.341498 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:35.331097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44468 (local address 127.0.97.2:41837)
0619 04:34:35.331651 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:35.331658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:35.331687 (+ 29us) server_negotiation.cc:407] Connection header received
0619 04:34:35.331743 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:35.331749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:35.331969 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:35.332131 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:35.333391 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:35.334442 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:35.335589 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:35.336124 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:35.336234 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:35.338604 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:35.338613 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:35.338627 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:35.338660 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:35.340889 (+ 2229us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 04:34:35.341190 (+ 301us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44468: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":273,"threads_started":1}
W20260619 04:34:35.341565 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 9368: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:41837: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 04:34:35.341603 9678 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44468: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 04:34:35.341722 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 9368
I20260619 04:34:35.354858 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 9499
I20260619 04:34:35.366051 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:35.355679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34098 (local address 127.0.97.3:38809)
0619 04:34:35.356223 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:35.356230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:35.356256 (+ 26us) server_negotiation.cc:407] Connection header received
0619 04:34:35.356313 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:35.356319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:35.356576 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:35.356731 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:35.357835 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:35.358824 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:35.359898 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:35.360587 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:35.360694 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:35.363113 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:35.363121 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:35.363136 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:35.363176 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:35.365414 (+ 2238us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 04:34:35.365713 (+ 299us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34098: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":286,"threads_started":1}
W20260619 04:34:35.366115 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 9499: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:38809: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 04:34:35.366164 9679 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34098: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 04:34:35.366242 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 9499
I20260619 04:34:35.378903 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 9148
I20260619 04:34:35.380224 9210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:35.495270 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 9148
2026-06-19T04:34:35Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20619 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T04:34:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:34:35Z Disabled control of system clock
I20260619 04:34:35.565929 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34059
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 04:34:35.873910 9684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:35.874184 9684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:35.874266 9684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:35.881309 9684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:34:35.881424 9684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:35.881469 9684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:35.881516 9684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:34:35.881557 9684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:34:35.890561 9684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34059
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:35.891779 9684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:35.893518 9684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:35.904817 9690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:35.904877 9689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:35.904855 9692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:35.905130 9684 server_base.cc:1034] running on GCE node
I20260619 04:34:35.906211 9684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:35.907608 9684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:35.908849 9684 hybrid_clock.cc:648] HybridClock initialized: now 1781843675908784 us; error 45 us; skew 500 ppm
I20260619 04:34:35.909205 9684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:34:35.911331 9684 webserver.cc:460] Webserver started at http://127.0.97.62:45647/ using document root <none> and password file <none>
I20260619 04:34:35.912004 9684 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:35.912099 9684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:35.912397 9684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:35.915217 9684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "96eacbc07e7548aa9c978e7e1d790f10"
format_stamp: "Formatted at 2026-06-19 04:34:35 on dist-test-slave-n3tq"
server_key: "d29897593ef91ed8bb40896fba4cf8f3"
server_key_iv: "550a2b95cfb473c5c6904aa608910de8"
server_key_version: "encryptionkey@0"
I20260619 04:34:35.915962 9684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "96eacbc07e7548aa9c978e7e1d790f10"
format_stamp: "Formatted at 2026-06-19 04:34:35 on dist-test-slave-n3tq"
server_key: "d29897593ef91ed8bb40896fba4cf8f3"
server_key_iv: "550a2b95cfb473c5c6904aa608910de8"
server_key_version: "encryptionkey@0"
I20260619 04:34:35.920820 9684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 04:34:35.924096 9698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:35.925597 9684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:34:35.925736 9684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "96eacbc07e7548aa9c978e7e1d790f10"
format_stamp: "Formatted at 2026-06-19 04:34:35 on dist-test-slave-n3tq"
server_key: "d29897593ef91ed8bb40896fba4cf8f3"
server_key_iv: "550a2b95cfb473c5c6904aa608910de8"
server_key_version: "encryptionkey@0"
I20260619 04:34:35.925916 9684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:35.967552 9684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:35.968354 9684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:35.968621 9684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:35.984477 9684 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34059
I20260619 04:34:35.984440 9749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34059 every 8 connection(s)
I20260619 04:34:35.985761 9684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:34:35.989446 9750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:35.994830 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 9684
I20260619 04:34:35.995018 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:34:35.995615 388 external_mini_cluster.cc:1428] Setting key f8b2bd7314d334f2916aa3459066d2d9
I20260619 04:34:35.999799 9750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10: Bootstrap starting.
I20260619 04:34:36.003619 9750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:36.004791 9750 log.cc:824] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10: Log is configured to *not* fsync() on all Append() calls
I20260619 04:34:36.007750 9750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10: No bootstrap required, opened a new log
I20260619 04:34:36.010586 9753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:35.997588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.0.97.62:34059)
0619 04:34:35.998592 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:35.998606 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:35.998653 (+ 47us) server_negotiation.cc:407] Connection header received
0619 04:34:36.000937 (+ 2284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:36.000973 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:36.001577 (+ 604us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:36.002023 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:36.002502 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:36.004347 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:36.005859 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:36.006760 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:36.006933 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:36.007907 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:36.007916 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:36.007934 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:34:36.008645 (+ 711us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:36.008734 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:34:36.008759 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:36.008863 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:36.009052 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 04:34:36.009346 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":317,"threads_started":1}
I20260619 04:34:36.016112 9750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96eacbc07e7548aa9c978e7e1d790f10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34059 } }
I20260619 04:34:36.016538 9750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:36.016636 9750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96eacbc07e7548aa9c978e7e1d790f10, State: Initialized, Role: FOLLOWER
I20260619 04:34:36.017189 9750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96eacbc07e7548aa9c978e7e1d790f10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34059 } }
I20260619 04:34:36.017401 9750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:34:36.017534 9750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:34:36.017669 9750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:36.019865 9750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96eacbc07e7548aa9c978e7e1d790f10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34059 } }
I20260619 04:34:36.020325 9750 leader_election.cc:302] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96eacbc07e7548aa9c978e7e1d790f10; no voters:
I20260619 04:34:36.020864 9750 leader_election.cc:288] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:34:36.020987 9755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:34:36.021555 9755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10 [term 1 LEADER]: Becoming Leader. State: Replica: 96eacbc07e7548aa9c978e7e1d790f10, State: Running, Role: LEADER
I20260619 04:34:36.022004 9755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96eacbc07e7548aa9c978e7e1d790f10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34059 } }
I20260619 04:34:36.022711 9750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:34:36.024528 9756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96eacbc07e7548aa9c978e7e1d790f10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96eacbc07e7548aa9c978e7e1d790f10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34059 } } }
I20260619 04:34:36.024535 9757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96eacbc07e7548aa9c978e7e1d790f10. Latest consensus state: current_term: 1 leader_uuid: "96eacbc07e7548aa9c978e7e1d790f10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96eacbc07e7548aa9c978e7e1d790f10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34059 } } }
I20260619 04:34:36.024945 9756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10 [sys.catalog]: This master's current role is: LEADER
I20260619 04:34:36.024982 9757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10 [sys.catalog]: This master's current role is: LEADER
I20260619 04:34:36.026372 9761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:34:36.031383 9761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:34:36.040709 9761 catalog_manager.cc:1347] Generated new cluster ID: 09db0e303c714ce4b7e30d484cb7fd03
I20260619 04:34:36.040822 9761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:34:36.062119 9761 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:34:36.063396 9761 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:34:36.073488 9761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96eacbc07e7548aa9c978e7e1d790f10: Generated new TSK 0
I20260619 04:34:36.074256 9761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:34:36.082387 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34059
--builtin_ntp_servers=127.0.97.20:41959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 04:34:36.390885 9774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:36.391150 9774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:36.391269 9774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:36.398506 9774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:36.398623 9774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:36.398767 9774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:34:36.407814 9774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:36.409096 9774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:36.410832 9774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:36.422904 9779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:36.422932 9780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:36.422962 9782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:36.423818 9774 server_base.cc:1034] running on GCE node
I20260619 04:34:36.424368 9774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:36.425256 9774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:36.426461 9774 hybrid_clock.cc:648] HybridClock initialized: now 1781843676426398 us; error 43 us; skew 500 ppm
I20260619 04:34:36.426746 9774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:34:36.428840 9774 webserver.cc:460] Webserver started at http://127.0.97.1:46277/ using document root <none> and password file <none>
I20260619 04:34:36.429502 9774 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:36.429597 9774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:36.429884 9774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:36.432687 9774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "29c04f2cb871449ca475786fa17d7395"
format_stamp: "Formatted at 2026-06-19 04:34:36 on dist-test-slave-n3tq"
server_key: "80ebe054f05a8bd95945ef4cb751ef35"
server_key_iv: "f3b673d5246f45b1923c808408201b84"
server_key_version: "encryptionkey@0"
I20260619 04:34:36.433436 9774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "29c04f2cb871449ca475786fa17d7395"
format_stamp: "Formatted at 2026-06-19 04:34:36 on dist-test-slave-n3tq"
server_key: "80ebe054f05a8bd95945ef4cb751ef35"
server_key_iv: "f3b673d5246f45b1923c808408201b84"
server_key_version: "encryptionkey@0"
I20260619 04:34:36.438202 9774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 04:34:36.441435 9788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:36.442678 9774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:34:36.442818 9774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "29c04f2cb871449ca475786fa17d7395"
format_stamp: "Formatted at 2026-06-19 04:34:36 on dist-test-slave-n3tq"
server_key: "80ebe054f05a8bd95945ef4cb751ef35"
server_key_iv: "f3b673d5246f45b1923c808408201b84"
server_key_version: "encryptionkey@0"
I20260619 04:34:36.442998 9774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:36.462308 9774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:36.463106 9774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:36.463371 9774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:36.464257 9774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:34:36.465673 9774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:36.465772 9774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:36.465876 9774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:36.465945 9774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:36.494498 9774 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41065
I20260619 04:34:36.494514 9900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41065 every 8 connection(s)
I20260619 04:34:36.495731 9774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:34:36.501746 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 9774
I20260619 04:34:36.501932 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:34:36.502496 388 external_mini_cluster.cc:1428] Setting key aac1ca7eda70a1f3736fc5669d7bc51f
I20260619 04:34:36.505059 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34059
--builtin_ntp_servers=127.0.97.20:41959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 04:34:36.507699 9753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:36.498058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43899 (local address 127.0.97.62:34059)
0619 04:34:36.498249 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:36.498255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:36.499643 (+ 1388us) server_negotiation.cc:407] Connection header received
0619 04:34:36.501352 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:36.501358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:36.501441 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:36.501567 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:36.502575 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:36.503409 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:36.505014 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:36.505562 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:36.505683 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:36.507008 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:36.507014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:36.507018 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:34:36.507037 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:36.507073 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:34:36.507086 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:36.507161 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:36.507335 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 04:34:36.507417 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 04:34:36.508673 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:36.498636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34059 (local address 127.0.97.1:43899)
0619 04:34:36.499502 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:36.499542 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:36.501093 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:36.501731 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:36.501739 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:36.501829 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:36.502422 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:36.502438 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:36.503556 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:36.503559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:36.504883 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:36.504894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:36.505780 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:36.505785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:36.505983 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:36.506704 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:34:36.506707 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:36.506744 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:34:36.506824 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:36.507169 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:36.507175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:36.507181 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 04:34:36.507325 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 04:34:36.507606 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":274,"threads_started":1}
I20260619 04:34:36.509956 9901 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34059
I20260619 04:34:36.510218 9901 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:36.510923 9901 heartbeater.cc:511] Master 127.0.97.62:34059 requested a full tablet report, sending...
I20260619 04:34:36.513123 9715 ts_manager.cc:194] Registered new tserver with Master: 29c04f2cb871449ca475786fa17d7395 (127.0.97.1:41065)
I20260619 04:34:36.514973 9715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:43899
W20260619 04:34:36.820953 9904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:36.821213 9904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:36.821293 9904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:36.828390 9904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:36.828541 9904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:36.828665 9904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:34:36.837726 9904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:36.838976 9904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:36.840797 9904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:36.853494 9909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:36.853525 9910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:36.853569 9912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:36.854597 9904 server_base.cc:1034] running on GCE node
I20260619 04:34:36.855199 9904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:36.856127 9904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:36.857324 9904 hybrid_clock.cc:648] HybridClock initialized: now 1781843676857273 us; error 47 us; skew 500 ppm
I20260619 04:34:36.857643 9904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:34:36.859748 9904 webserver.cc:460] Webserver started at http://127.0.97.2:44961/ using document root <none> and password file <none>
I20260619 04:34:36.860423 9904 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:36.860566 9904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:36.860872 9904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:36.863682 9904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "055d442252c341dea6a55ac79c072b2b"
format_stamp: "Formatted at 2026-06-19 04:34:36 on dist-test-slave-n3tq"
server_key: "08df6e1dea8adf4f37af0999fd74c1e0"
server_key_iv: "816dd9b74b43529715cdcf84975265e3"
server_key_version: "encryptionkey@0"
I20260619 04:34:36.864444 9904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "055d442252c341dea6a55ac79c072b2b"
format_stamp: "Formatted at 2026-06-19 04:34:36 on dist-test-slave-n3tq"
server_key: "08df6e1dea8adf4f37af0999fd74c1e0"
server_key_iv: "816dd9b74b43529715cdcf84975265e3"
server_key_version: "encryptionkey@0"
I20260619 04:34:36.869714 9904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260619 04:34:36.873049 9918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:36.874269 9904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:34:36.874418 9904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "055d442252c341dea6a55ac79c072b2b"
format_stamp: "Formatted at 2026-06-19 04:34:36 on dist-test-slave-n3tq"
server_key: "08df6e1dea8adf4f37af0999fd74c1e0"
server_key_iv: "816dd9b74b43529715cdcf84975265e3"
server_key_version: "encryptionkey@0"
I20260619 04:34:36.874624 9904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:36.898923 9904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:36.899753 9904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:36.900005 9904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:36.900955 9904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:34:36.902351 9904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:36.902443 9904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:36.902557 9904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:36.902624 9904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:36.931375 9904 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37403
I20260619 04:34:36.931442 10030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37403 every 8 connection(s)
I20260619 04:34:36.932644 9904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:34:36.937530 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 9904
I20260619 04:34:36.937693 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:34:36.938076 388 external_mini_cluster.cc:1428] Setting key 22f54437c0a0f5651d8523b3d75eebca
I20260619 04:34:36.940838 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34059
--builtin_ntp_servers=127.0.97.20:41959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 04:34:36.944978 9753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:36.935034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48859 (local address 127.0.97.62:34059)
0619 04:34:36.935221 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:36.935228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:36.936673 (+ 1445us) server_negotiation.cc:407] Connection header received
0619 04:34:36.938457 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:36.938463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:36.938554 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:36.938656 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:36.939760 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:36.940806 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:36.942423 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:36.942929 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:36.943047 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:36.944239 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:36.944245 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:36.944249 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:34:36.944269 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:36.944304 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:34:36.944317 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:36.944391 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:36.944592 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 04:34:36.944677 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 04:34:36.945979 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:36.935628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34059 (local address 127.0.97.2:48859)
0619 04:34:36.936538 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:36.936574 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:36.938196 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:36.938910 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:36.938919 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:36.939018 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:36.939604 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:36.939629 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:36.941077 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:36.941081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:36.942290 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:36.942301 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:36.943111 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:36.943116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:36.943314 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:36.943941 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:34:36.943943 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:36.943992 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:34:36.944075 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:36.944399 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:36.944405 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:36.944410 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 04:34:36.944572 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 04:34:36.944867 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":270,"threads_started":1}
I20260619 04:34:36.947479 10031 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34059
I20260619 04:34:36.947763 10031 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:36.948657 10031 heartbeater.cc:511] Master 127.0.97.62:34059 requested a full tablet report, sending...
I20260619 04:34:36.950098 9715 ts_manager.cc:194] Registered new tserver with Master: 055d442252c341dea6a55ac79c072b2b (127.0.97.2:37403)
I20260619 04:34:36.950873 9715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:48859
W20260619 04:34:37.283036 10034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:37.283336 10034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:37.283465 10034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:37.290834 10034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:37.290951 10034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:37.291105 10034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:34:37.300325 10034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:37.301666 10034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:37.303381 10034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:37.315758 10040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:37.315864 10042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:37.315989 10039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:37.316807 10034 server_base.cc:1034] running on GCE node
I20260619 04:34:37.317395 10034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:37.318284 10034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:37.319500 10034 hybrid_clock.cc:648] HybridClock initialized: now 1781843677319448 us; error 41 us; skew 500 ppm
I20260619 04:34:37.319806 10034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:34:37.321971 10034 webserver.cc:460] Webserver started at http://127.0.97.3:33699/ using document root <none> and password file <none>
I20260619 04:34:37.322633 10034 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:37.322726 10034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:37.323020 10034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:37.325841 10034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "0f64110e2b9943e3b8afde0645bc6924"
format_stamp: "Formatted at 2026-06-19 04:34:37 on dist-test-slave-n3tq"
server_key: "6bcd26519ffcb34a4ad3f60732500495"
server_key_iv: "c1881e4254e6669823fe43897956ff0c"
server_key_version: "encryptionkey@0"
I20260619 04:34:37.326596 10034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0f64110e2b9943e3b8afde0645bc6924"
format_stamp: "Formatted at 2026-06-19 04:34:37 on dist-test-slave-n3tq"
server_key: "6bcd26519ffcb34a4ad3f60732500495"
server_key_iv: "c1881e4254e6669823fe43897956ff0c"
server_key_version: "encryptionkey@0"
I20260619 04:34:37.331457 10034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 04:34:37.334637 10048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:37.335925 10034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 04:34:37.336072 10034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "0f64110e2b9943e3b8afde0645bc6924"
format_stamp: "Formatted at 2026-06-19 04:34:37 on dist-test-slave-n3tq"
server_key: "6bcd26519ffcb34a4ad3f60732500495"
server_key_iv: "c1881e4254e6669823fe43897956ff0c"
server_key_version: "encryptionkey@0"
I20260619 04:34:37.336256 10034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:37.372792 10034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:37.373636 10034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:37.373880 10034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:37.374748 10034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:34:37.376111 10034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:37.376199 10034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:37.376304 10034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:37.376375 10034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:37.403240 10034 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36295
I20260619 04:34:37.403319 10160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36295 every 8 connection(s)
I20260619 04:34:37.404489 10034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:34:37.407095 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 10034
I20260619 04:34:37.407274 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:34:37.407747 388 external_mini_cluster.cc:1428] Setting key 41e70c7bb5d6996060f9dc2d187a2ebf
I20260619 04:34:37.416527 9753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:37.406907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44499 (local address 127.0.97.62:34059)
0619 04:34:37.407061 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:37.407066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:37.408872 (+ 1806us) server_negotiation.cc:407] Connection header received
0619 04:34:37.410991 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:37.410996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:37.411060 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:37.411137 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:37.412111 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:37.412846 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:37.414259 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:37.414655 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:37.414742 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:37.415875 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:37.415880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:37.415884 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:34:37.415903 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:37.415932 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:34:37.415944 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:37.416008 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:37.416172 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 04:34:37.416242 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 04:34:37.417559 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:37.407633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34059 (local address 127.0.97.3:44499)
0619 04:34:37.408723 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:37.408765 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:37.410711 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:37.411316 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:37.411324 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:37.411445 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:37.411993 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:37.412008 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:37.412959 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:37.412963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:37.414156 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:37.414167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:37.414809 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:37.414813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:37.415007 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:37.415606 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:34:37.415608 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:37.415645 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:34:37.415723 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:37.416015 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:37.416023 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:37.416028 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 04:34:37.416160 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 04:34:37.416487 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":312,"threads_started":1}
I20260619 04:34:37.418819 10161 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34059
I20260619 04:34:37.419137 10161 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:37.419885 10161 heartbeater.cc:511] Master 127.0.97.62:34059 requested a full tablet report, sending...
I20260619 04:34:37.421273 9715 ts_manager.cc:194] Registered new tserver with Master: 0f64110e2b9943e3b8afde0645bc6924 (127.0.97.3:36295)
I20260619 04:34:37.422145 9715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:44499
I20260619 04:34:37.422972 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 04:34:37.428614 9753 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:37.427211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54950 (local address 127.0.97.62:34059)
0619 04:34:37.427395 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:37.427402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:37.427772 (+ 370us) server_negotiation.cc:407] Connection header received
0619 04:34:37.427905 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:37.427909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:37.427969 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:37.428056 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:37.428344 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54950: BlockingRecv error: recv got EOF from 127.0.0.1:54950 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 04:34:37.429248 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 9774
I20260619 04:34:37.436225 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:37.430050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40892 (local address 127.0.97.1:41065)
0619 04:34:37.430539 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:37.430546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:37.430569 (+ 23us) server_negotiation.cc:407] Connection header received
0619 04:34:37.430618 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:37.430623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:37.430817 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:37.430936 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:37.431321 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:37.432250 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:37.433444 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:37.433989 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:37.434098 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:37.434973 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:37.434981 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:37.434994 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:34:37.435541 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:37.435585 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:34:37.435603 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:37.435687 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:37.435790 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 04:34:37.435893 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":270,"threads_started":1}
I20260619 04:34:37.437507 9896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:37.516533 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 9774
I20260619 04:34:37.530335 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 9904
I20260619 04:34:37.537447 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:37.531157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52878 (local address 127.0.97.2:37403)
0619 04:34:37.531635 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:37.531642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:37.531669 (+ 27us) server_negotiation.cc:407] Connection header received
0619 04:34:37.531727 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:37.531733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:37.531964 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:37.532090 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:37.532418 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:37.533408 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:37.534570 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:37.535102 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:37.535194 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:37.536094 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:37.536101 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:37.536116 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:34:37.536790 (+ 674us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:37.536837 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:34:37.536855 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:37.536934 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:37.537032 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 04:34:37.537148 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":221,"threads_started":1}
I20260619 04:34:37.538743 10026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:37.616865 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 9904
I20260619 04:34:37.629637 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 10034
I20260619 04:34:37.637200 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:37.630597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56294 (local address 127.0.97.3:36295)
0619 04:34:37.631243 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:37.631250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:37.631279 (+ 29us) server_negotiation.cc:407] Connection header received
0619 04:34:37.631337 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:37.631343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:37.631574 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:37.631696 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:37.632074 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:37.633157 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:37.634398 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:37.634927 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:37.635028 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:37.635868 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:37.635876 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:37.635889 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:34:37.636447 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:37.636534 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:34:37.636553 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:37.636629 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:37.636743 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 04:34:37.636847 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":358,"threads_started":1}
I20260619 04:34:37.638401 10156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:37.750609 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 10034
I20260619 04:34:37.770687 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 9684
I20260619 04:34:37.771986 9745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:37.871184 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 9684
2026-06-19T04:34:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2366 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T04:34:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:34:37Z Disabled control of system clock
I20260619 04:34:37.933683 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36817
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 04:34:38.243090 10180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:38.243368 10180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:38.243455 10180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:38.250411 10180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:34:38.250519 10180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:38.250552 10180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:38.250607 10180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:34:38.250648 10180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:34:38.259469 10180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36817
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:38.260663 10180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:38.262396 10180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:38.273339 10185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:38.273584 10188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:38.273604 10186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:38.274474 10180 server_base.cc:1034] running on GCE node
I20260619 04:34:38.275236 10180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:38.276634 10180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:38.277851 10180 hybrid_clock.cc:648] HybridClock initialized: now 1781843678277800 us; error 41 us; skew 500 ppm
I20260619 04:34:38.278192 10180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:34:38.280294 10180 webserver.cc:460] Webserver started at http://127.0.97.62:41737/ using document root <none> and password file <none>
I20260619 04:34:38.281010 10180 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:38.281103 10180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:38.281418 10180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:38.284265 10180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "f77fe84acf384d42975ae9c56b39e09c"
format_stamp: "Formatted at 2026-06-19 04:34:38 on dist-test-slave-n3tq"
server_key: "6b0ed941e542cdbe25ea24a30c470cf2"
server_key_iv: "2338808f7b68199f89ccc88654b467c3"
server_key_version: "encryptionkey@0"
I20260619 04:34:38.285080 10180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "f77fe84acf384d42975ae9c56b39e09c"
format_stamp: "Formatted at 2026-06-19 04:34:38 on dist-test-slave-n3tq"
server_key: "6b0ed941e542cdbe25ea24a30c470cf2"
server_key_iv: "2338808f7b68199f89ccc88654b467c3"
server_key_version: "encryptionkey@0"
I20260619 04:34:38.290019 10180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 04:34:38.293365 10194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:38.294721 10180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:34:38.294885 10180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "f77fe84acf384d42975ae9c56b39e09c"
format_stamp: "Formatted at 2026-06-19 04:34:38 on dist-test-slave-n3tq"
server_key: "6b0ed941e542cdbe25ea24a30c470cf2"
server_key_iv: "2338808f7b68199f89ccc88654b467c3"
server_key_version: "encryptionkey@0"
I20260619 04:34:38.295094 10180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:38.300815 10180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:38.301534 10180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:38.301790 10180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:38.317931 10180 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36817
I20260619 04:34:38.317950 10245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36817 every 8 connection(s)
I20260619 04:34:38.319193 10180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:34:38.322218 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 10180
I20260619 04:34:38.322381 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:34:38.322710 10246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:38.322999 388 external_mini_cluster.cc:1428] Setting key 4124f36bcf68e7940fc00e89266d26d8
I20260619 04:34:38.331475 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:38.324805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46400 (local address 127.0.97.62:36817)
0619 04:34:38.325786 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:38.325796 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:38.325837 (+ 41us) server_negotiation.cc:407] Connection header received
0619 04:34:38.328049 (+ 2212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:38.328072 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:38.328555 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:38.328897 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:38.329971 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:38.329977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:38.329990 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:34:38.330003 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:38.330063 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:34:38.330077 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:38.330240 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:38.330330 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 04:34:38.330607 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":356,"threads_started":1}
I20260619 04:34:38.334976 10246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c: Bootstrap starting.
I20260619 04:34:38.337824 10246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:38.338877 10246 log.cc:824] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c: Log is configured to *not* fsync() on all Append() calls
I20260619 04:34:38.341404 10246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c: No bootstrap required, opened a new log
I20260619 04:34:38.348001 10246 raft_consensus.cc:348] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77fe84acf384d42975ae9c56b39e09c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36817 } }
I20260619 04:34:38.348311 10246 raft_consensus.cc:374] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:38.348380 10246 raft_consensus.cc:729] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f77fe84acf384d42975ae9c56b39e09c, State: Initialized, Role: FOLLOWER
I20260619 04:34:38.348858 10246 consensus_queue.cc:260] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77fe84acf384d42975ae9c56b39e09c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36817 } }
I20260619 04:34:38.349049 10246 raft_consensus.cc:388] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:34:38.349138 10246 raft_consensus.cc:482] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:34:38.349248 10246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:38.351363 10246 raft_consensus.cc:504] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77fe84acf384d42975ae9c56b39e09c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36817 } }
I20260619 04:34:38.351766 10246 leader_election.cc:302] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f77fe84acf384d42975ae9c56b39e09c; no voters:
I20260619 04:34:38.352264 10246 leader_election.cc:288] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:34:38.352425 10251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:34:38.353055 10251 raft_consensus.cc:686] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c [term 1 LEADER]: Becoming Leader. State: Replica: f77fe84acf384d42975ae9c56b39e09c, State: Running, Role: LEADER
I20260619 04:34:38.353525 10251 consensus_queue.cc:237] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77fe84acf384d42975ae9c56b39e09c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36817 } }
I20260619 04:34:38.354121 10246 sys_catalog.cc:564] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:34:38.356078 10253 sys_catalog.cc:455] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f77fe84acf384d42975ae9c56b39e09c. Latest consensus state: current_term: 1 leader_uuid: "f77fe84acf384d42975ae9c56b39e09c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77fe84acf384d42975ae9c56b39e09c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36817 } } }
I20260619 04:34:38.356073 10252 sys_catalog.cc:455] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f77fe84acf384d42975ae9c56b39e09c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77fe84acf384d42975ae9c56b39e09c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36817 } } }
I20260619 04:34:38.356509 10253 sys_catalog.cc:458] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c [sys.catalog]: This master's current role is: LEADER
I20260619 04:34:38.357340 10252 sys_catalog.cc:458] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c [sys.catalog]: This master's current role is: LEADER
I20260619 04:34:38.361150 10259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:34:38.366089 10259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:34:38.375597 10259 catalog_manager.cc:1347] Generated new cluster ID: d62a8cd4bc964e68b1231e7861c1020e
I20260619 04:34:38.375699 10259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:34:38.391754 10259 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:34:38.392146 10259 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:34:38.399644 10259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f77fe84acf384d42975ae9c56b39e09c: Generated new TSK 0
I20260619 04:34:38.400390 10259 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:34:38.412380 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36817
--builtin_ntp_servers=127.0.97.20:36517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 04:34:38.726490 10270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:38.726775 10270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:38.726899 10270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:38.734063 10270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:38.734175 10270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:38.734323 10270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:34:38.743335 10270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:38.744685 10270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:38.746438 10270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:38.758481 10275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:38.758606 10278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:38.758504 10276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:38.759371 10270 server_base.cc:1034] running on GCE node
I20260619 04:34:38.759938 10270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:38.760929 10270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:38.762135 10270 hybrid_clock.cc:648] HybridClock initialized: now 1781843678762085 us; error 41 us; skew 500 ppm
I20260619 04:34:38.762454 10270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:34:38.764582 10270 webserver.cc:460] Webserver started at http://127.0.97.1:33059/ using document root <none> and password file <none>
I20260619 04:34:38.765232 10270 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:38.765326 10270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:38.765616 10270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:38.768306 10270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "f7d6188f797547fa9312d554f935b3d2"
format_stamp: "Formatted at 2026-06-19 04:34:38 on dist-test-slave-n3tq"
server_key: "6734d6fd9d0081c8a64755ee7e2216d4"
server_key_iv: "e33b15bb0fb39e7921b02dab65d58399"
server_key_version: "encryptionkey@0"
I20260619 04:34:38.769063 10270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f7d6188f797547fa9312d554f935b3d2"
format_stamp: "Formatted at 2026-06-19 04:34:38 on dist-test-slave-n3tq"
server_key: "6734d6fd9d0081c8a64755ee7e2216d4"
server_key_iv: "e33b15bb0fb39e7921b02dab65d58399"
server_key_version: "encryptionkey@0"
I20260619 04:34:38.773795 10270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260619 04:34:38.777014 10284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:38.778273 10270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:34:38.778410 10270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "f7d6188f797547fa9312d554f935b3d2"
format_stamp: "Formatted at 2026-06-19 04:34:38 on dist-test-slave-n3tq"
server_key: "6734d6fd9d0081c8a64755ee7e2216d4"
server_key_iv: "e33b15bb0fb39e7921b02dab65d58399"
server_key_version: "encryptionkey@0"
I20260619 04:34:38.778616 10270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:38.783789 10270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:38.784557 10270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:38.784780 10270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:38.785607 10270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:34:38.786854 10270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:38.786942 10270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:38.787045 10270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:38.787110 10270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:38.813617 10270 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44495
I20260619 04:34:38.813660 10396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44495 every 8 connection(s)
I20260619 04:34:38.814849 10270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:34:38.822141 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:38.817179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38969 (local address 127.0.97.62:36817)
0619 04:34:38.817395 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:38.817401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:38.818765 (+ 1364us) server_negotiation.cc:407] Connection header received
0619 04:34:38.820300 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:38.820303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:38.820355 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:38.820443 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:38.821618 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:38.821623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:38.821626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:34:38.821640 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:38.821669 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:34:38.821680 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:38.821770 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:38.821894 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 04:34:38.821938 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 04:34:38.823014 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:38.817757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36817 (local address 127.0.97.1:38969)
0619 04:34:38.818618 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:38.818652 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:38.820076 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:38.820686 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:38.820695 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:38.820786 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:38.821358 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:34:38.821361 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:38.821395 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:34:38.821476 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:38.821752 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:38.821757 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:38.821762 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 04:34:38.821948 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 04:34:38.822147 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":257,"threads_started":1}
I20260619 04:34:38.824321 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 10270
I20260619 04:34:38.824398 10397 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36817
I20260619 04:34:38.824530 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:34:38.824717 10397 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:38.825171 388 external_mini_cluster.cc:1428] Setting key 4d1efcd7b72aabe28c6d7fc454083cfe
I20260619 04:34:38.825259 10397 heartbeater.cc:511] Master 127.0.97.62:36817 requested a full tablet report, sending...
I20260619 04:34:38.827286 10210 ts_manager.cc:194] Registered new tserver with Master: f7d6188f797547fa9312d554f935b3d2 (127.0.97.1:44495)
I20260619 04:34:38.827718 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36817
--builtin_ntp_servers=127.0.97.20:36517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 04:34:39.134575 10400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:39.134857 10400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:39.134977 10400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:39.142012 10400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:39.142123 10400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:39.142269 10400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:34:39.151211 10400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:39.152474 10400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:39.154175 10400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:39.166602 10405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:39.166581 10406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:39.166530 10408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:39.167330 10400 server_base.cc:1034] running on GCE node
I20260619 04:34:39.167917 10400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:39.168876 10400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:39.170064 10400 hybrid_clock.cc:648] HybridClock initialized: now 1781843679170017 us; error 38 us; skew 500 ppm
I20260619 04:34:39.170352 10400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:34:39.172338 10400 webserver.cc:460] Webserver started at http://127.0.97.2:40729/ using document root <none> and password file <none>
I20260619 04:34:39.173022 10400 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:39.173122 10400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:39.173396 10400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:39.177639 10400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "202be21fdba24621a395bdf2abaddab2"
format_stamp: "Formatted at 2026-06-19 04:34:39 on dist-test-slave-n3tq"
server_key: "2020b54d11934f1c9287e77ced10177a"
server_key_iv: "fe2e6c3415652ee31c8598ae73c68e9a"
server_key_version: "encryptionkey@0"
I20260619 04:34:39.178406 10400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "202be21fdba24621a395bdf2abaddab2"
format_stamp: "Formatted at 2026-06-19 04:34:39 on dist-test-slave-n3tq"
server_key: "2020b54d11934f1c9287e77ced10177a"
server_key_iv: "fe2e6c3415652ee31c8598ae73c68e9a"
server_key_version: "encryptionkey@0"
I20260619 04:34:39.183401 10400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:34:39.186687 10414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:39.187934 10400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 04:34:39.188069 10400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "202be21fdba24621a395bdf2abaddab2"
format_stamp: "Formatted at 2026-06-19 04:34:39 on dist-test-slave-n3tq"
server_key: "2020b54d11934f1c9287e77ced10177a"
server_key_iv: "fe2e6c3415652ee31c8598ae73c68e9a"
server_key_version: "encryptionkey@0"
I20260619 04:34:39.188266 10400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:39.193449 10400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:39.194161 10400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:39.194381 10400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:39.195215 10400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:34:39.196470 10400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:39.196574 10400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:39.196668 10400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:39.196733 10400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:39.224718 10400 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35061
I20260619 04:34:39.224764 10526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35061 every 8 connection(s)
I20260619 04:34:39.225973 10400 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:34:39.226379 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 10400
I20260619 04:34:39.226507 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:34:39.227021 388 external_mini_cluster.cc:1428] Setting key 0a0a9f673bb96536b8adcd56c73a3d50
I20260619 04:34:39.230111 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36817
--builtin_ntp_servers=127.0.97.20:36517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 04:34:39.234187 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:39.228636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57503 (local address 127.0.97.62:36817)
0619 04:34:39.228826 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:39.228832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:39.230467 (+ 1635us) server_negotiation.cc:407] Connection header received
0619 04:34:39.232088 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:39.232093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:39.232155 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:39.232245 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:39.233502 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:39.233508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:39.233513 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:34:39.233532 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:39.233567 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:34:39.233578 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:39.233675 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:39.233849 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 04:34:39.233912 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 04:34:39.235213 10529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:39.229323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36817 (local address 127.0.97.2:57503)
0619 04:34:39.230304 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:39.230356 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:39.231834 (+ 1478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:39.232496 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:39.232507 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:39.232612 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:39.233227 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:34:39.233229 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:39.233265 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:34:39.233345 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:39.233657 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:39.233666 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:39.233673 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 04:34:39.233943 (+ 270us) client_negotiation.cc:240] Negotiation successful
0619 04:34:39.234262 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":298,"threads_started":1}
I20260619 04:34:39.236568 10527 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36817
I20260619 04:34:39.236860 10527 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:39.237498 10527 heartbeater.cc:511] Master 127.0.97.62:36817 requested a full tablet report, sending...
I20260619 04:34:39.238966 10210 ts_manager.cc:194] Registered new tserver with Master: 202be21fdba24621a395bdf2abaddab2 (127.0.97.2:35061)
W20260619 04:34:39.542305 10530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:39.542574 10530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:39.542665 10530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:39.549618 10530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:39.549724 10530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:39.549831 10530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:34:39.558840 10530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:39.560038 10530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:39.561844 10530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:39.573910 10535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:39.573855 10536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:39.574288 10538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:39.574978 10530 server_base.cc:1034] running on GCE node
I20260619 04:34:39.575549 10530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:39.576498 10530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:39.577728 10530 hybrid_clock.cc:648] HybridClock initialized: now 1781843679577672 us; error 45 us; skew 500 ppm
I20260619 04:34:39.578049 10530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:34:39.580150 10530 webserver.cc:460] Webserver started at http://127.0.97.3:43609/ using document root <none> and password file <none>
I20260619 04:34:39.580893 10530 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:39.580997 10530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:39.581283 10530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:39.584085 10530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "b04f053d94fc4c1aa9617acd434146fd"
format_stamp: "Formatted at 2026-06-19 04:34:39 on dist-test-slave-n3tq"
server_key: "d59dcbc59472a2c7945a920f7d2a6fa3"
server_key_iv: "6f5b4e7bbd1676feccc70c6692cf36c6"
server_key_version: "encryptionkey@0"
I20260619 04:34:39.584848 10530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b04f053d94fc4c1aa9617acd434146fd"
format_stamp: "Formatted at 2026-06-19 04:34:39 on dist-test-slave-n3tq"
server_key: "d59dcbc59472a2c7945a920f7d2a6fa3"
server_key_iv: "6f5b4e7bbd1676feccc70c6692cf36c6"
server_key_version: "encryptionkey@0"
I20260619 04:34:39.589618 10530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 04:34:39.592888 10544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:39.594101 10530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 04:34:39.594244 10530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "b04f053d94fc4c1aa9617acd434146fd"
format_stamp: "Formatted at 2026-06-19 04:34:39 on dist-test-slave-n3tq"
server_key: "d59dcbc59472a2c7945a920f7d2a6fa3"
server_key_iv: "6f5b4e7bbd1676feccc70c6692cf36c6"
server_key_version: "encryptionkey@0"
I20260619 04:34:39.594451 10530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:39.599659 10530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:39.600400 10530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:39.600648 10530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:39.601496 10530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:34:39.602743 10530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:39.602833 10530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:39.602936 10530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:39.603005 10530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:39.631166 10530 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46253
I20260619 04:34:39.631211 10656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46253 every 8 connection(s)
I20260619 04:34:39.632436 10530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:34:39.639696 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:39.634843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46481 (local address 127.0.97.62:36817)
0619 04:34:39.635029 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:39.635034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:39.636355 (+ 1321us) server_negotiation.cc:407] Connection header received
0619 04:34:39.637951 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:39.637955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:39.638011 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:39.638098 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:39.639176 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:39.639180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:39.639183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:34:39.639196 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:39.639224 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:34:39.639235 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:39.639327 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:39.639461 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 04:34:39.639506 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 04:34:39.640580 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:39.635364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36817 (local address 127.0.97.3:46481)
0619 04:34:39.636215 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:39.636252 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:39.637735 (+ 1483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:39.638278 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:39.638286 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:39.638375 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:39.638926 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:34:39.638928 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:39.638962 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:34:39.639040 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:39.639303 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:39.639308 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:39.639313 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 04:34:39.639524 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 04:34:39.639706 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":244,"threads_started":1}
I20260619 04:34:39.641779 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 10530
I20260619 04:34:39.641813 10657 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36817
I20260619 04:34:39.641927 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:34:39.642077 10657 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:39.642365 388 external_mini_cluster.cc:1428] Setting key ffb7e1efbe5888edbe70b82557004589
I20260619 04:34:39.642627 10657 heartbeater.cc:511] Master 127.0.97.62:36817 requested a full tablet report, sending...
I20260619 04:34:39.644022 10210 ts_manager.cc:194] Registered new tserver with Master: b04f053d94fc4c1aa9617acd434146fd (127.0.97.3:46253)
I20260619 04:34:39.645179 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 04:34:39.650503 10249 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:34:39.649179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46416 (local address 127.0.97.62:36817)
0619 04:34:39.649339 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:39.649345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:39.649764 (+ 419us) server_negotiation.cc:407] Connection header received
0619 04:34:39.649895 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:39.649898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:39.649953 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:39.650021 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:39.650266 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46416: BlockingRecv error: recv got EOF from 127.0.0.1:46416 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 04:34:39.651140 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 10270
I20260619 04:34:39.654294 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:39.652012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43328 (local address 127.0.97.1:44495)
0619 04:34:39.652492 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:39.652508 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:39.652536 (+ 28us) server_negotiation.cc:407] Connection header received
0619 04:34:39.652590 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:39.652596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:39.652783 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:39.652898 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:39.653717 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:39.653725 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:39.653739 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:34:39.653761 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:39.653803 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:34:39.653817 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:39.653915 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:39.653991 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 04:34:39.654071 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":215,"threads_started":1}
I20260619 04:34:39.655632 10392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:39.733964 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 10270
I20260619 04:34:39.746917 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 10400
I20260619 04:34:39.750159 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:39.747717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.0.97.2:35061)
0619 04:34:39.748158 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:39.748165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:39.748190 (+ 25us) server_negotiation.cc:407] Connection header received
0619 04:34:39.748249 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:39.748254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:39.748496 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:39.748641 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:39.749513 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:39.749519 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:39.749532 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:34:39.749544 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:39.749581 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:34:39.749595 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:39.749712 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:39.749819 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 04:34:39.749920 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":192,"threads_started":1}
I20260619 04:34:39.751286 10522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:39.834205 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 10400
I20260619 04:34:39.846707 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 10530
I20260619 04:34:39.850258 10670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:39.847502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48562 (local address 127.0.97.3:46253)
0619 04:34:39.848071 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:39.848078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:39.848106 (+ 28us) server_negotiation.cc:407] Connection header received
0619 04:34:39.848164 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:39.848171 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:39.848381 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:39.848539 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:39.849570 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:39.849576 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:39.849588 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:34:39.849599 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:39.849637 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:34:39.849650 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:39.849799 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:39.849872 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 04:34:39.849973 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":258,"threads_started":1}
I20260619 04:34:39.851406 10652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:39.930632 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 10530
I20260619 04:34:39.943145 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 10180
I20260619 04:34:39.944242 10241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:40.041843 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 10180
2026-06-19T04:34:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2171 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:34:40 dist-test-slave-n3tq krb5kdc[10675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:34:40 dist-test-slave-n3tq krb5kdc[10675](info): set up 2 sockets
Jun 19 04:34:40 dist-test-slave-n3tq krb5kdc[10675](info): commencing operation
krb5kdc: starting...
W20260619 04:34:41.053047 388 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:34:41 dist-test-slave-n3tq krb5kdc[10675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:34:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:34:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:34:41.256284 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43343
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:34:41.560639 10688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:41.560933 10688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:41.561044 10688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:41.568023 10688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:34:41.568136 10688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:41.568207 10688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:41.568279 10688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:34:41.568336 10688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:34:41.577754 10688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:41.578981 10688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:41.580227 10688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:41.590734 10696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:41.590767 10693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:41.590700 10694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:41.591725 10688 server_base.cc:1034] running on GCE node
I20260619 04:34:41.592406 10688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:41.593735 10688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:41.594918 10688 hybrid_clock.cc:648] HybridClock initialized: now 1781843681594867 us; error 44 us; skew 500 ppm
I20260619 04:34:41.595219 10688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:41 dist-test-slave-n3tq krb5kdc[10675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:41.598053 10688 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:34:41.599311 10688 webserver.cc:460] Webserver started at http://127.0.97.62:45191/ using document root <none> and password file <none>
I20260619 04:34:41.599908 10688 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:41.600006 10688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:41.600231 10688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:41.603060 10688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "d32af981a3f04d9dbd9069e1858e51fd"
format_stamp: "Formatted at 2026-06-19 04:34:41 on dist-test-slave-n3tq"
server_key: "2b3d2ba54d0c00094da793851355da2e"
server_key_iv: "97f78858247a47fb5345019544ba0ef3"
server_key_version: "encryptionkey@0"
I20260619 04:34:41.603713 10688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "d32af981a3f04d9dbd9069e1858e51fd"
format_stamp: "Formatted at 2026-06-19 04:34:41 on dist-test-slave-n3tq"
server_key: "2b3d2ba54d0c00094da793851355da2e"
server_key_iv: "97f78858247a47fb5345019544ba0ef3"
server_key_version: "encryptionkey@0"
I20260619 04:34:41.608376 10688 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 04:34:41.611377 10703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:41.612591 10688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260619 04:34:41.612728 10688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "d32af981a3f04d9dbd9069e1858e51fd"
format_stamp: "Formatted at 2026-06-19 04:34:41 on dist-test-slave-n3tq"
server_key: "2b3d2ba54d0c00094da793851355da2e"
server_key_iv: "97f78858247a47fb5345019544ba0ef3"
server_key_version: "encryptionkey@0"
I20260619 04:34:41.612872 10688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:41.627275 10688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:41.630748 10688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:41.630956 10688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:41.646849 10688 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43343
I20260619 04:34:41.646870 10754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43343 every 8 connection(s)
I20260619 04:34:41.648111 10688 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:34:41.651583 10755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:41.655062 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 10688
I20260619 04:34:41.655251 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:34:41.655989 388 external_mini_cluster.cc:1428] Setting key 0117018f67262a23678db9af397ff004
I20260619 04:34:41.663853 10755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd: Bootstrap starting.
I20260619 04:34:41.667308 10755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:41.668551 10755 log.cc:824] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd: Log is configured to *not* fsync() on all Append() calls
Jun 19 04:34:41 dist-test-slave-n3tq krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:34:41.671568 10755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd: No bootstrap required, opened a new log
I20260619 04:34:41.677858 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:41.657935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.0.97.62:43343)
0619 04:34:41.658625 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:41.658636 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:41.658673 (+ 37us) server_negotiation.cc:407] Connection header received
0619 04:34:41.660157 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:41.660177 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:41.660609 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:41.660918 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:41.662504 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:41.663792 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:41.665509 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:41.666170 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:41.666301 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:41.670889 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:41.670896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:41.670911 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:41.670948 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:41.673993 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:41.674759 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:41.674766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:41.674772 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:41.674936 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:41.675296 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:41.675299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:41.675301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:41.675791 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:41.676016 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:41.676407 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 04:34:41.676801 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":310,"threads_started":1}
I20260619 04:34:41.679710 10755 raft_consensus.cc:348] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d32af981a3f04d9dbd9069e1858e51fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43343 } }
I20260619 04:34:41.680042 10755 raft_consensus.cc:374] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:41.680140 10755 raft_consensus.cc:729] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d32af981a3f04d9dbd9069e1858e51fd, State: Initialized, Role: FOLLOWER
I20260619 04:34:41.680732 10755 consensus_queue.cc:260] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d32af981a3f04d9dbd9069e1858e51fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43343 } }
I20260619 04:34:41.680929 10755 raft_consensus.cc:388] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:34:41.681047 10755 raft_consensus.cc:482] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:34:41.681185 10755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:41.683367 10755 raft_consensus.cc:504] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d32af981a3f04d9dbd9069e1858e51fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43343 } }
I20260619 04:34:41.683799 10755 leader_election.cc:302] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d32af981a3f04d9dbd9069e1858e51fd; no voters:
I20260619 04:34:41.684301 10755 leader_election.cc:288] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:34:41.684401 10760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:34:41.684996 10760 raft_consensus.cc:686] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd [term 1 LEADER]: Becoming Leader. State: Replica: d32af981a3f04d9dbd9069e1858e51fd, State: Running, Role: LEADER
I20260619 04:34:41.685456 10760 consensus_queue.cc:237] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d32af981a3f04d9dbd9069e1858e51fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43343 } }
I20260619 04:34:41.686084 10755 sys_catalog.cc:564] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:34:41.687839 10762 sys_catalog.cc:455] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d32af981a3f04d9dbd9069e1858e51fd. Latest consensus state: current_term: 1 leader_uuid: "d32af981a3f04d9dbd9069e1858e51fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d32af981a3f04d9dbd9069e1858e51fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43343 } } }
I20260619 04:34:41.688199 10762 sys_catalog.cc:458] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd [sys.catalog]: This master's current role is: LEADER
I20260619 04:34:41.687839 10761 sys_catalog.cc:455] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d32af981a3f04d9dbd9069e1858e51fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d32af981a3f04d9dbd9069e1858e51fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43343 } } }
I20260619 04:34:41.688808 10761 sys_catalog.cc:458] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd [sys.catalog]: This master's current role is: LEADER
I20260619 04:34:41.693982 10767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:34:41.698537 10767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:34:41.707603 10767 catalog_manager.cc:1347] Generated new cluster ID: c5fe975db03f4de28579d241ab9a9dd8
I20260619 04:34:41.707716 10767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:34:41.720476 10767 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:34:41.722260 10767 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:34:41.731387 10767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd: Generated new TSK 0
I20260619 04:34:41.732192 10767 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 04:34:41.793758 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43343
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:34:42.128695 10781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:42.128959 10781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:42.129099 10781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:42.136232 10781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:42.136340 10781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:42.136510 10781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:34:42.145820 10781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:42.147014 10781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:42.148278 10781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:42.160029 10789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:42.160071 10787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:42.160194 10786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:42.161000 10781 server_base.cc:1034] running on GCE node
I20260619 04:34:42.161532 10781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:42.162446 10781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:42.163625 10781 hybrid_clock.cc:648] HybridClock initialized: now 1781843682163584 us; error 37 us; skew 500 ppm
I20260619 04:34:42.163884 10781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:42 dist-test-slave-n3tq krb5kdc[10675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:42.166697 10781 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:34:42.167965 10781 webserver.cc:460] Webserver started at http://127.0.97.1:43749/ using document root <none> and password file <none>
I20260619 04:34:42.168630 10781 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:42.168726 10781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:42.168982 10781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:42.171643 10781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "ee4ccf72494b4646b14245431b0badc8"
format_stamp: "Formatted at 2026-06-19 04:34:42 on dist-test-slave-n3tq"
server_key: "9a4c2d3c714b61fd99b132708230705b"
server_key_iv: "251c70f54abc9a304ec7624a5353769b"
server_key_version: "encryptionkey@0"
I20260619 04:34:42.172309 10781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ee4ccf72494b4646b14245431b0badc8"
format_stamp: "Formatted at 2026-06-19 04:34:42 on dist-test-slave-n3tq"
server_key: "9a4c2d3c714b61fd99b132708230705b"
server_key_iv: "251c70f54abc9a304ec7624a5353769b"
server_key_version: "encryptionkey@0"
I20260619 04:34:42.176831 10781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 04:34:42.179939 10796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:42.181174 10781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 04:34:42.181314 10781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "ee4ccf72494b4646b14245431b0badc8"
format_stamp: "Formatted at 2026-06-19 04:34:42 on dist-test-slave-n3tq"
server_key: "9a4c2d3c714b61fd99b132708230705b"
server_key_iv: "251c70f54abc9a304ec7624a5353769b"
server_key_version: "encryptionkey@0"
I20260619 04:34:42.181468 10781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:42.198256 10781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:42.201958 10781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:42.202240 10781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:42.203073 10781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:34:42.204286 10781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:42.204352 10781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:42.204442 10781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:42.204522 10781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 04:34:42.232710 10781 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35495
I20260619 04:34:42.232761 10908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35495 every 8 connection(s)
I20260619 04:34:42.233952 10781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:34:42.237121 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 10781
I20260619 04:34:42.237301 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:34:42.238103 388 external_mini_cluster.cc:1428] Setting key b06607165b614bd7b39b185aa81a5a71
Jun 19 04:34:42 dist-test-slave-n3tq krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:34:42.254710 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:42.236310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36659 (local address 127.0.97.62:43343)
0619 04:34:42.236754 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:42.236761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:42.237951 (+ 1190us) server_negotiation.cc:407] Connection header received
0619 04:34:42.240012 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:42.240016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:42.240078 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:42.240162 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:42.242017 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:42.242715 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:42.244719 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:42.245124 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:42.245225 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:42.249060 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:42.249067 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:42.249072 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:42.249110 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:42.251913 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:42.252640 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:42.252645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:42.252648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:42.252774 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:42.253128 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:42.253132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:42.253135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:42.253433 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:42.253575 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:42.254198 (+ 623us) server_negotiation.cc:299] Negotiation successful
0619 04:34:42.254395 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":230,"threads_started":1}
I20260619 04:34:42.255524 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:42.236927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43343 (local address 127.0.97.1:36659)
0619 04:34:42.237769 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:42.237829 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:42.239740 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:42.240420 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:42.240432 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:42.241099 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:42.241848 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:42.241871 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:42.242859 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:42.242864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:42.244554 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:42.244568 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:42.245328 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:42.245333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:42.245594 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:42.246249 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:42.246275 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:42.248827 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:42.252067 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:42.252074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:42.252088 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:42.252503 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:42.252865 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:42.252868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:42.252870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:42.253033 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:42.253554 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:42.253560 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:42.253858 (+ 298us) client_negotiation.cc:769] Sending connection context
0619 04:34:42.254072 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 04:34:42.254448 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":301,"threads_started":1}
I20260619 04:34:42.256930 10909 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43343
I20260619 04:34:42.257261 10909 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:42.258160 10909 heartbeater.cc:511] Master 127.0.97.62:43343 requested a full tablet report, sending...
I20260619 04:34:42.260262 10720 ts_manager.cc:194] Registered new tserver with Master: ee4ccf72494b4646b14245431b0badc8 (127.0.97.1:35495)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 04:34:42.261737 10720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 04:34:42.287551 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43343
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:34:42.590432 10915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:42.590706 10915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:42.590823 10915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:42.597822 10915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:42.597941 10915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:42.598101 10915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:34:42.608031 10915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:42.609524 10915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:42.611121 10915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:42.624435 10921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:42.624478 10920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:42.625051 10923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:42.625519 10915 server_base.cc:1034] running on GCE node
I20260619 04:34:42.626049 10915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:42.626891 10915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:42.628079 10915 hybrid_clock.cc:648] HybridClock initialized: now 1781843682628037 us; error 31 us; skew 500 ppm
I20260619 04:34:42.628355 10915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:42 dist-test-slave-n3tq krb5kdc[10675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:42.631117 10915 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:34:42.632395 10915 webserver.cc:460] Webserver started at http://127.0.97.2:33805/ using document root <none> and password file <none>
I20260619 04:34:42.633028 10915 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:42.633132 10915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:42.633388 10915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:42.635978 10915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "0623a426159d463fbf10bea7ff042b0a"
format_stamp: "Formatted at 2026-06-19 04:34:42 on dist-test-slave-n3tq"
server_key: "53e2033f6a5ee7ac5bdeef0d6bf3d97b"
server_key_iv: "60973c8893eb76e0abae6d27d535883a"
server_key_version: "encryptionkey@0"
I20260619 04:34:42.636693 10915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0623a426159d463fbf10bea7ff042b0a"
format_stamp: "Formatted at 2026-06-19 04:34:42 on dist-test-slave-n3tq"
server_key: "53e2033f6a5ee7ac5bdeef0d6bf3d97b"
server_key_iv: "60973c8893eb76e0abae6d27d535883a"
server_key_version: "encryptionkey@0"
I20260619 04:34:42.641292 10915 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 04:34:42.644384 10930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:42.645720 10915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:34:42.645854 10915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "0623a426159d463fbf10bea7ff042b0a"
format_stamp: "Formatted at 2026-06-19 04:34:42 on dist-test-slave-n3tq"
server_key: "53e2033f6a5ee7ac5bdeef0d6bf3d97b"
server_key_iv: "60973c8893eb76e0abae6d27d535883a"
server_key_version: "encryptionkey@0"
I20260619 04:34:42.646013 10915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:42.671344 10915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:42.674903 10915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:42.675114 10915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:42.675925 10915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:34:42.677205 10915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:42.677275 10915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:42.677359 10915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:42.677398 10915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:42.705492 10915 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35883
I20260619 04:34:42.705535 11042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35883 every 8 connection(s)
I20260619 04:34:42.706687 10915 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:34:42.716732 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 10915
I20260619 04:34:42.716897 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:34:42.717320 388 external_mini_cluster.cc:1428] Setting key 79c829154074cd8671f4c52741d9f351
Jun 19 04:34:42 dist-test-slave-n3tq krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:34:42.724908 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:42.709019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49269 (local address 127.0.97.62:43343)
0619 04:34:42.709204 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:42.709210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:42.710302 (+ 1092us) server_negotiation.cc:407] Connection header received
0619 04:34:42.711959 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:42.711963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:42.712026 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:42.712112 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:42.713436 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:42.714129 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:42.715534 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:42.715927 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:42.716014 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:42.719447 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:42.719454 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:42.719459 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:42.719495 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:42.722285 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:42.722924 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:42.722929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:42.722932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:42.723062 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:42.723392 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:42.723396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:42.723399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:42.723691 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:42.723809 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:42.724387 (+ 578us) server_negotiation.cc:299] Negotiation successful
0619 04:34:42.724606 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 04:34:42.725723 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:42.709497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43343 (local address 127.0.97.2:49269)
0619 04:34:42.710155 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:42.710192 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:42.711735 (+ 1543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:42.712288 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:42.712295 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:42.712783 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:42.713302 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:42.713318 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:42.714236 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:42.714240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:42.715389 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:42.715400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:42.716084 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:42.716089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:42.716306 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:42.716990 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:42.717022 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:42.719240 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:42.722397 (+ 3157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:42.722403 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:42.722414 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:42.722831 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:42.723153 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:42.723156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:42.723158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:42.723313 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:42.723801 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:42.723806 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:42.724087 (+ 281us) client_negotiation.cc:769] Sending connection context
0619 04:34:42.724294 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 04:34:42.724672 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":244,"threads_started":1}
I20260619 04:34:42.727106 11043 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43343
I20260619 04:34:42.727370 11043 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:42.727991 11043 heartbeater.cc:511] Master 127.0.97.62:43343 requested a full tablet report, sending...
I20260619 04:34:42.729277 10720 ts_manager.cc:194] Registered new tserver with Master: 0623a426159d463fbf10bea7ff042b0a (127.0.97.2:35883)
I20260619 04:34:42.730031 10720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49269
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 04:34:42.762750 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43343
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:34:43.066457 11048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:34:43.066753 11048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:34:43.066875 11048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:34:43.073971 11048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:34:43.074090 11048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:34:43.074245 11048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:34:43.083719 11048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:34:43.084991 11048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:34:43.086318 11048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:34:43.098490 11054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:43.098539 11056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:34:43.099639 11053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:34:43.100239 11048 server_base.cc:1034] running on GCE node
I20260619 04:34:43.100903 11048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:34:43.101758 11048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:34:43.102942 11048 hybrid_clock.cc:648] HybridClock initialized: now 1781843683102896 us; error 38 us; skew 500 ppm
I20260619 04:34:43.103227 11048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:34:43 dist-test-slave-n3tq krb5kdc[10675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:34:43.106086 11048 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:34:43.107355 11048 webserver.cc:460] Webserver started at http://127.0.97.3:38027/ using document root <none> and password file <none>
I20260619 04:34:43.107990 11048 fs_manager.cc:359] Metadata directory not provided
I20260619 04:34:43.108096 11048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:34:43.108366 11048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:34:43.111114 11048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "d5bab5abc3724763bd421922c40d3b7e"
format_stamp: "Formatted at 2026-06-19 04:34:43 on dist-test-slave-n3tq"
server_key: "81523759d88b0b991575fa9a0bd9938e"
server_key_iv: "14983eb4331ae74253e231a878bb4636"
server_key_version: "encryptionkey@0"
I20260619 04:34:43.111814 11048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d5bab5abc3724763bd421922c40d3b7e"
format_stamp: "Formatted at 2026-06-19 04:34:43 on dist-test-slave-n3tq"
server_key: "81523759d88b0b991575fa9a0bd9938e"
server_key_iv: "14983eb4331ae74253e231a878bb4636"
server_key_version: "encryptionkey@0"
I20260619 04:34:43.116679 11048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 04:34:43.119889 11063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:43.121304 11048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:34:43.121454 11048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "d5bab5abc3724763bd421922c40d3b7e"
format_stamp: "Formatted at 2026-06-19 04:34:43 on dist-test-slave-n3tq"
server_key: "81523759d88b0b991575fa9a0bd9938e"
server_key_iv: "14983eb4331ae74253e231a878bb4636"
server_key_version: "encryptionkey@0"
I20260619 04:34:43.121652 11048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:34:43.133510 11048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:34:43.137465 11048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:34:43.137720 11048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:34:43.138628 11048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:34:43.139840 11048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:34:43.139916 11048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:43.140033 11048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:34:43.140095 11048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:34:43.168390 11048 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42465
I20260619 04:34:43.168430 11175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42465 every 8 connection(s)
I20260619 04:34:43.169591 11048 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:34:43.171713 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 11048
I20260619 04:34:43.171880 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:34:43.172348 388 external_mini_cluster.cc:1428] Setting key ab781d73f2a121b33f5fd0b021f3b9a4
Jun 19 04:34:43 dist-test-slave-n3tq krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:34:43.188016 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:43.171923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58323 (local address 127.0.97.62:43343)
0619 04:34:43.172129 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:43.172136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:43.173300 (+ 1164us) server_negotiation.cc:407] Connection header received
0619 04:34:43.175355 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:43.175360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:43.175438 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:43.175516 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:43.176940 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.177684 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.179157 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.179536 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.179644 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:43.182656 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:43.182660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:43.182664 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:43.182695 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:43.185218 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:43.185957 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:43.185961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:43.185963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:43.186072 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:43.186507 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:43.186511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:43.186514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:43.186818 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:43.186932 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:43.187627 (+ 695us) server_negotiation.cc:299] Negotiation successful
0619 04:34:43.187784 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 04:34:43.189090 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:43.172382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43343 (local address 127.0.97.3:58323)
0619 04:34:43.173160 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:43.173204 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:43.175070 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:43.175745 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:43.175753 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:43.176222 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:34:43.176796 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:43.176812 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.177815 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.177820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:43.179019 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:43.179031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.179728 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.179735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:43.179960 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:43.180526 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:34:43.180546 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:34:43.182481 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:34:43.185358 (+ 2877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:43.185366 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:43.185382 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:43.185811 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:43.186168 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:34:43.186171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:34:43.186174 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:34:43.186393 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:34:43.186977 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:34:43.186988 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:34:43.187304 (+ 316us) client_negotiation.cc:769] Sending connection context
0619 04:34:43.187512 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 04:34:43.187880 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":293,"threads_started":1}
I20260619 04:34:43.190219 11176 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43343
I20260619 04:34:43.190481 11176 heartbeater.cc:464] Registering TS with master...
I20260619 04:34:43.191121 11176 heartbeater.cc:511] Master 127.0.97.62:43343 requested a full tablet report, sending...
I20260619 04:34:43.192358 10720 ts_manager.cc:194] Registered new tserver with Master: d5bab5abc3724763bd421922c40d3b7e (127.0.97.3:42465)
I20260619 04:34:43.193132 10720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58323
I20260619 04:34:43.198977 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:34:43.214073 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:43.202933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.0.97.62:43343)
0619 04:34:43.203075 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:43.203080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:43.203473 (+ 393us) server_negotiation.cc:407] Connection header received
0619 04:34:43.203649 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:43.203653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:43.203708 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:43.203780 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:43.205236 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.205917 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.207062 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.207443 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.207546 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:43.209590 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:43.209595 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:43.209598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:43.209626 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:43.211762 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:43.212330 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:43.212336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:43.212340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:43.212446 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:43.212785 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:43.212790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:43.212794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:43.213074 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:43.213282 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:43.213595 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 04:34:43.213791 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 04:34:43.218402 10720 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:34:43.226406 10720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:34:43.259505 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:43.238632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50624 (local address 127.0.97.1:35495)
0619 04:34:43.251169 (+ 12537us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:43.251177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:43.251201 (+ 24us) server_negotiation.cc:407] Connection header received
0619 04:34:43.251273 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:43.251278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:43.251428 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:43.251595 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:43.252278 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.253841 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.256535 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.257967 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.258191 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:43.258437 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:43.259006 (+ 569us) server_negotiation.cc:299] Negotiation successful
0619 04:34:43.259181 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12383,"thread_start_us":12257,"threads_started":1}
I20260619 04:34:43.259507 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:43.238204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35495 (local address 127.0.0.1:50624)
0619 04:34:43.250485 (+ 12281us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:43.250501 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:43.250602 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:43.251671 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:43.251675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:43.251698 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:43.252059 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:43.252081 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.253974 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.253979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:43.256360 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:43.256374 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.258564 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.258571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:43.258754 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:43.258810 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 04:34:43.258940 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 04:34:43.259128 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12175,"mutex_wait_us":33,"thread_start_us":12059,"threads_started":1}
I20260619 04:34:43.259735 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:43.238041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35818 (local address 127.0.97.2:35883)
0619 04:34:43.249177 (+ 11136us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:43.249187 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:43.250023 (+ 836us) server_negotiation.cc:407] Connection header received
0619 04:34:43.250485 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:43.250494 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:43.250758 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:43.250923 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:43.251822 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.253449 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.256860 (+ 3411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.258305 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.258445 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:43.258672 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:43.259138 (+ 466us) server_negotiation.cc:299] Negotiation successful
0619 04:34:43.259297 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10930,"thread_start_us":11119,"threads_started":1}
I20260619 04:34:43.260285 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:43.237865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35883 (local address 127.0.0.1:35818)
0619 04:34:43.249876 (+ 12011us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:43.249918 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:43.250181 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:43.251208 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:43.251214 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:43.251245 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:43.251617 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:43.251630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.253626 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.253630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:43.256721 (+ 3091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:43.256734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.258809 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.258815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:43.258966 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:43.258994 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 04:34:43.259766 (+ 772us) client_negotiation.cc:240] Negotiation successful
0619 04:34:43.259899 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11781,"mutex_wait_us":659,"thread_start_us":11649,"threads_started":1}
I20260619 04:34:43.261309 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:43.251960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42465 (local address 127.0.0.1:35568)
0619 04:34:43.254419 (+ 2459us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:43.254437 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:43.254561 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:43.255041 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:43.255045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:43.255067 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:43.255439 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:43.255450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.256767 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.256771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:43.258482 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:43.258494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.260556 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.260562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:43.260714 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:43.260744 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 04:34:43.260821 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 04:34:43.260939 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2321,"thread_start_us":2189,"threads_started":1}
I20260619 04:34:43.262202 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:43.251402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.0.97.3:42465)
0619 04:34:43.253752 (+ 2350us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:43.253760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:43.254546 (+ 786us) server_negotiation.cc:407] Connection header received
0619 04:34:43.254690 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:43.254698 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:43.254872 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:43.255047 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:43.255626 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.256644 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.258672 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.260348 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.260497 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:43.261664 (+ 1167us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:43.261737 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 04:34:43.261907 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2200,"thread_start_us":2055,"threads_started":1}
I20260619 04:34:43.263566 10844 tablet_service.cc:1467] Processing CreateTablet for tablet 252f4dfc0c284cd59b7361de730ab47f (DEFAULT_TABLE table=test-table [id=2f71bc90186c4f399f7ffbb97e68cb67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:34:43.263566 10978 tablet_service.cc:1467] Processing CreateTablet for tablet 252f4dfc0c284cd59b7361de730ab47f (DEFAULT_TABLE table=test-table [id=2f71bc90186c4f399f7ffbb97e68cb67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:34:43.264597 10909 heartbeater.cc:503] Master 127.0.97.62:43343 was elected leader, sending a full tablet report...
I20260619 04:34:43.264901 10978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 252f4dfc0c284cd59b7361de730ab47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:43.264909 10844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 252f4dfc0c284cd59b7361de730ab47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:43.265518 11111 tablet_service.cc:1467] Processing CreateTablet for tablet 252f4dfc0c284cd59b7361de730ab47f (DEFAULT_TABLE table=test-table [id=2f71bc90186c4f399f7ffbb97e68cb67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:34:43.266796 11111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 252f4dfc0c284cd59b7361de730ab47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:34:43.275889 11194 tablet_bootstrap.cc:492] T 252f4dfc0c284cd59b7361de730ab47f P ee4ccf72494b4646b14245431b0badc8: Bootstrap starting.
I20260619 04:34:43.278434 11195 tablet_bootstrap.cc:492] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a: Bootstrap starting.
I20260619 04:34:43.279424 11194 tablet_bootstrap.cc:654] T 252f4dfc0c284cd59b7361de730ab47f P ee4ccf72494b4646b14245431b0badc8: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:43.280030 11196 tablet_bootstrap.cc:492] T 252f4dfc0c284cd59b7361de730ab47f P d5bab5abc3724763bd421922c40d3b7e: Bootstrap starting.
I20260619 04:34:43.281257 11194 log.cc:824] T 252f4dfc0c284cd59b7361de730ab47f P ee4ccf72494b4646b14245431b0badc8: Log is configured to *not* fsync() on all Append() calls
I20260619 04:34:43.281883 11195 tablet_bootstrap.cc:654] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:43.282891 11195 log.cc:824] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a: Log is configured to *not* fsync() on all Append() calls
I20260619 04:34:43.283321 11196 tablet_bootstrap.cc:654] T 252f4dfc0c284cd59b7361de730ab47f P d5bab5abc3724763bd421922c40d3b7e: Neither blocks nor log segments found. Creating new log.
I20260619 04:34:43.284305 11194 tablet_bootstrap.cc:492] T 252f4dfc0c284cd59b7361de730ab47f P ee4ccf72494b4646b14245431b0badc8: No bootstrap required, opened a new log
I20260619 04:34:43.284493 11196 log.cc:824] T 252f4dfc0c284cd59b7361de730ab47f P d5bab5abc3724763bd421922c40d3b7e: Log is configured to *not* fsync() on all Append() calls
I20260619 04:34:43.284601 11194 ts_tablet_manager.cc:1397] T 252f4dfc0c284cd59b7361de730ab47f P ee4ccf72494b4646b14245431b0badc8: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 04:34:43.286034 11195 tablet_bootstrap.cc:492] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a: No bootstrap required, opened a new log
I20260619 04:34:43.286320 11195 ts_tablet_manager.cc:1397] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 04:34:43.286895 11196 tablet_bootstrap.cc:492] T 252f4dfc0c284cd59b7361de730ab47f P d5bab5abc3724763bd421922c40d3b7e: No bootstrap required, opened a new log
I20260619 04:34:43.287120 11196 ts_tablet_manager.cc:1397] T 252f4dfc0c284cd59b7361de730ab47f P d5bab5abc3724763bd421922c40d3b7e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260619 04:34:43.294389 11196 raft_consensus.cc:348] T 252f4dfc0c284cd59b7361de730ab47f P d5bab5abc3724763bd421922c40d3b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0623a426159d463fbf10bea7ff042b0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35883 } } peers { permanent_uuid: "ee4ccf72494b4646b14245431b0badc8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35495 } } peers { permanent_uuid: "d5bab5abc3724763bd421922c40d3b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } }
I20260619 04:34:43.294730 11196 raft_consensus.cc:374] T 252f4dfc0c284cd59b7361de730ab47f P d5bab5abc3724763bd421922c40d3b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:43.294520 11194 raft_consensus.cc:348] T 252f4dfc0c284cd59b7361de730ab47f P ee4ccf72494b4646b14245431b0badc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0623a426159d463fbf10bea7ff042b0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35883 } } peers { permanent_uuid: "ee4ccf72494b4646b14245431b0badc8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35495 } } peers { permanent_uuid: "d5bab5abc3724763bd421922c40d3b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } }
I20260619 04:34:43.294821 11196 raft_consensus.cc:729] T 252f4dfc0c284cd59b7361de730ab47f P d5bab5abc3724763bd421922c40d3b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5bab5abc3724763bd421922c40d3b7e, State: Initialized, Role: FOLLOWER
I20260619 04:34:43.294929 11194 raft_consensus.cc:374] T 252f4dfc0c284cd59b7361de730ab47f P ee4ccf72494b4646b14245431b0badc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:43.295019 11194 raft_consensus.cc:729] T 252f4dfc0c284cd59b7361de730ab47f P ee4ccf72494b4646b14245431b0badc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4ccf72494b4646b14245431b0badc8, State: Initialized, Role: FOLLOWER
I20260619 04:34:43.295429 11196 consensus_queue.cc:260] T 252f4dfc0c284cd59b7361de730ab47f P d5bab5abc3724763bd421922c40d3b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0623a426159d463fbf10bea7ff042b0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35883 } } peers { permanent_uuid: "ee4ccf72494b4646b14245431b0badc8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35495 } } peers { permanent_uuid: "d5bab5abc3724763bd421922c40d3b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } }
I20260619 04:34:43.295564 11194 consensus_queue.cc:260] T 252f4dfc0c284cd59b7361de730ab47f P ee4ccf72494b4646b14245431b0badc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0623a426159d463fbf10bea7ff042b0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35883 } } peers { permanent_uuid: "ee4ccf72494b4646b14245431b0badc8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35495 } } peers { permanent_uuid: "d5bab5abc3724763bd421922c40d3b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } }
I20260619 04:34:43.296164 11195 raft_consensus.cc:348] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0623a426159d463fbf10bea7ff042b0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35883 } } peers { permanent_uuid: "ee4ccf72494b4646b14245431b0badc8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35495 } } peers { permanent_uuid: "d5bab5abc3724763bd421922c40d3b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } }
I20260619 04:34:43.296617 11195 raft_consensus.cc:374] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:34:43.296715 11195 raft_consensus.cc:729] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0623a426159d463fbf10bea7ff042b0a, State: Initialized, Role: FOLLOWER
I20260619 04:34:43.297405 11194 ts_tablet_manager.cc:1428] T 252f4dfc0c284cd59b7361de730ab47f P ee4ccf72494b4646b14245431b0badc8: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260619 04:34:43.297284 11195 consensus_queue.cc:260] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0623a426159d463fbf10bea7ff042b0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35883 } } peers { permanent_uuid: "ee4ccf72494b4646b14245431b0badc8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35495 } } peers { permanent_uuid: "d5bab5abc3724763bd421922c40d3b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } }
I20260619 04:34:43.298444 11043 heartbeater.cc:503] Master 127.0.97.62:43343 was elected leader, sending a full tablet report...
I20260619 04:34:43.298972 11176 heartbeater.cc:503] Master 127.0.97.62:43343 was elected leader, sending a full tablet report...
I20260619 04:34:43.299511 11196 ts_tablet_manager.cc:1428] T 252f4dfc0c284cd59b7361de730ab47f P d5bab5abc3724763bd421922c40d3b7e: Time spent starting tablet: real 0.012s user 0.011s sys 0.002s
I20260619 04:34:43.300594 11195 ts_tablet_manager.cc:1428] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260619 04:34:43.410363 11202 raft_consensus.cc:482] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:34:43.410646 11202 raft_consensus.cc:504] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0623a426159d463fbf10bea7ff042b0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35883 } } peers { permanent_uuid: "ee4ccf72494b4646b14245431b0badc8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35495 } } peers { permanent_uuid: "d5bab5abc3724763bd421922c40d3b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } }
I20260619 04:34:43.412257 11202 leader_election.cc:288] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee4ccf72494b4646b14245431b0badc8 (127.0.97.1:35495), d5bab5abc3724763bd421922c40d3b7e (127.0.97.3:42465)
I20260619 04:34:43.418375 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:43.412653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35117 (local address 127.0.97.1:35495)
0619 04:34:43.412832 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:43.412838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:43.413365 (+ 527us) server_negotiation.cc:407] Connection header received
0619 04:34:43.413549 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:43.413555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:43.413633 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:43.413717 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:43.414264 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.414954 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.416719 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.417566 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.417691 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:43.417788 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:43.418036 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 04:34:43.418153 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 04:34:43.418469 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:43.412653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35495 (local address 127.0.97.2:35117)
0619 04:34:43.413260 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:43.413279 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:43.413420 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:43.413764 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:43.413768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:43.413801 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:43.414126 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:43.414134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.415071 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.415082 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:43.416572 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:43.416585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.417768 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.417774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:43.417901 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:43.417924 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 04:34:43.418004 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 04:34:43.418134 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":317,"threads_started":1}
I20260619 04:34:43.418792 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:43.413480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41857 (local address 127.0.97.3:42465)
0619 04:34:43.413672 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:43.413678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:43.414038 (+ 360us) server_negotiation.cc:407] Connection header received
0619 04:34:43.414183 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:43.414187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:43.414243 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:43.414312 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:34:43.414772 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.415475 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.417213 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.417962 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.418083 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:43.418176 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:43.418457 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 04:34:43.418572 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 04:34:43.418850 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:43.413339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42465 (local address 127.0.97.2:41857)
0619 04:34:43.413929 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0619 04:34:43.413941 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 04:34:43.414056 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:34:43.414334 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:34:43.414338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:34:43.414359 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:34:43.414638 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:43.414646 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.415622 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.415627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:43.417071 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:34:43.417082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.418162 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.418168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:34:43.418320 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:43.418341 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 04:34:43.418413 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 04:34:43.418539 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":308,"threads_started":1}
I20260619 04:34:43.419451 10864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252f4dfc0c284cd59b7361de730ab47f" candidate_uuid: "0623a426159d463fbf10bea7ff042b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4ccf72494b4646b14245431b0badc8" is_pre_election: true
I20260619 04:34:43.419703 11131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252f4dfc0c284cd59b7361de730ab47f" candidate_uuid: "0623a426159d463fbf10bea7ff042b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5bab5abc3724763bd421922c40d3b7e" is_pre_election: true
I20260619 04:34:43.419911 10864 raft_consensus.cc:2454] T 252f4dfc0c284cd59b7361de730ab47f P ee4ccf72494b4646b14245431b0badc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0623a426159d463fbf10bea7ff042b0a in term 0.
I20260619 04:34:43.420181 11131 raft_consensus.cc:2454] T 252f4dfc0c284cd59b7361de730ab47f P d5bab5abc3724763bd421922c40d3b7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0623a426159d463fbf10bea7ff042b0a in term 0.
I20260619 04:34:43.420568 10932 leader_election.cc:302] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0623a426159d463fbf10bea7ff042b0a, ee4ccf72494b4646b14245431b0badc8; no voters:
I20260619 04:34:43.420970 11202 raft_consensus.cc:2781] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260619 04:34:43.420985 11177 tablet.cc:2369] T 252f4dfc0c284cd59b7361de730ab47f P d5bab5abc3724763bd421922c40d3b7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:34:43.421104 11202 raft_consensus.cc:482] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:34:43.421211 11202 raft_consensus.cc:3037] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:43.423265 11202 raft_consensus.cc:504] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0623a426159d463fbf10bea7ff042b0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35883 } } peers { permanent_uuid: "ee4ccf72494b4646b14245431b0badc8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35495 } } peers { permanent_uuid: "d5bab5abc3724763bd421922c40d3b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } }
I20260619 04:34:43.423784 11202 leader_election.cc:288] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a [CANDIDATE]: Term 1 election: Requested vote from peers ee4ccf72494b4646b14245431b0badc8 (127.0.97.1:35495), d5bab5abc3724763bd421922c40d3b7e (127.0.97.3:42465)
I20260619 04:34:43.424194 10864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252f4dfc0c284cd59b7361de730ab47f" candidate_uuid: "0623a426159d463fbf10bea7ff042b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4ccf72494b4646b14245431b0badc8"
I20260619 04:34:43.424260 11131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252f4dfc0c284cd59b7361de730ab47f" candidate_uuid: "0623a426159d463fbf10bea7ff042b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5bab5abc3724763bd421922c40d3b7e"
I20260619 04:34:43.424367 10864 raft_consensus.cc:3037] T 252f4dfc0c284cd59b7361de730ab47f P ee4ccf72494b4646b14245431b0badc8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:43.424423 11131 raft_consensus.cc:3037] T 252f4dfc0c284cd59b7361de730ab47f P d5bab5abc3724763bd421922c40d3b7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:34:43.426577 11131 raft_consensus.cc:2454] T 252f4dfc0c284cd59b7361de730ab47f P d5bab5abc3724763bd421922c40d3b7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0623a426159d463fbf10bea7ff042b0a in term 1.
I20260619 04:34:43.426687 10864 raft_consensus.cc:2454] T 252f4dfc0c284cd59b7361de730ab47f P ee4ccf72494b4646b14245431b0badc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0623a426159d463fbf10bea7ff042b0a in term 1.
I20260619 04:34:43.426908 10932 leader_election.cc:302] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0623a426159d463fbf10bea7ff042b0a, d5bab5abc3724763bd421922c40d3b7e; no voters:
I20260619 04:34:43.427198 11202 raft_consensus.cc:2781] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:34:43.427597 11202 raft_consensus.cc:686] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a [term 1 LEADER]: Becoming Leader. State: Replica: 0623a426159d463fbf10bea7ff042b0a, State: Running, Role: LEADER
I20260619 04:34:43.428081 11202 consensus_queue.cc:237] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0623a426159d463fbf10bea7ff042b0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35883 } } peers { permanent_uuid: "ee4ccf72494b4646b14245431b0badc8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35495 } } peers { permanent_uuid: "d5bab5abc3724763bd421922c40d3b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } }
I20260619 04:34:43.432245 10720 catalog_manager.cc:5495] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0623a426159d463fbf10bea7ff042b0a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0623a426159d463fbf10bea7ff042b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0623a426159d463fbf10bea7ff042b0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4ccf72494b4646b14245431b0badc8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5bab5abc3724763bd421922c40d3b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 } health_report { overall_health: UNKNOWN } } }
I20260619 04:34:43.451923 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:43.446839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35832 (local address 127.0.97.2:35883)
0619 04:34:43.447027 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:43.447034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:43.447105 (+ 71us) server_negotiation.cc:407] Connection header received
0619 04:34:43.447320 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:43.447324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:43.447397 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:43.447519 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:34:43.447868 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.448620 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.449882 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.450319 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.450424 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:43.450861 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:34:43.451369 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:34:43.451502 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:43.451579 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 04:34:43.451661 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 04:34:43.458424 11044 tablet.cc:2369] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:34:43.459874 10864 raft_consensus.cc:1261] T 252f4dfc0c284cd59b7361de730ab47f P ee4ccf72494b4646b14245431b0badc8 [term 1 FOLLOWER]: Refusing update from remote peer 0623a426159d463fbf10bea7ff042b0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:34:43.459873 11131 raft_consensus.cc:1261] T 252f4dfc0c284cd59b7361de730ab47f P d5bab5abc3724763bd421922c40d3b7e [term 1 FOLLOWER]: Refusing update from remote peer 0623a426159d463fbf10bea7ff042b0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:34:43.460819 11205 consensus_queue.cc:1035] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4ccf72494b4646b14245431b0badc8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:34:43.461153 11202 consensus_queue.cc:1035] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5bab5abc3724763bd421922c40d3b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:34:43.472834 11209 mvcc.cc:205] Tried to move back new op lower bound from 7298431727441248256 to 7298431727325298688. Current Snapshot: MvccSnapshot[applied={T|T < 7298431727441248256}]
I20260619 04:34:43.486044 10717 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:34:43.486390 10717 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:34:43.489828 10717 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d32af981a3f04d9dbd9069e1858e51fd: Sending DeleteTablet for 3 replicas of tablet 252f4dfc0c284cd59b7361de730ab47f
I20260619 04:34:43.490942 10978 tablet_service.cc:1514] Processing DeleteTablet for tablet 252f4dfc0c284cd59b7361de730ab47f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:34:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35818
I20260619 04:34:43.491209 11111 tablet_service.cc:1514] Processing DeleteTablet for tablet 252f4dfc0c284cd59b7361de730ab47f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:34:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35568
I20260619 04:34:43.491371 10844 tablet_service.cc:1514] Processing DeleteTablet for tablet 252f4dfc0c284cd59b7361de730ab47f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:34:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50624
I20260619 04:34:43.493386 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 10781
I20260619 04:34:43.502476 11221 tablet_replica.cc:330] T 252f4dfc0c284cd59b7361de730ab47f P ee4ccf72494b4646b14245431b0badc8: stopping tablet replica
I20260619 04:34:43.503098 11221 raft_consensus.cc:2229] T 252f4dfc0c284cd59b7361de730ab47f P ee4ccf72494b4646b14245431b0badc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:34:43.503739 11221 raft_consensus.cc:2258] T 252f4dfc0c284cd59b7361de730ab47f P ee4ccf72494b4646b14245431b0badc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:34:43.504587 11220 tablet_replica.cc:330] T 252f4dfc0c284cd59b7361de730ab47f P d5bab5abc3724763bd421922c40d3b7e: stopping tablet replica
I20260619 04:34:43.505038 11220 raft_consensus.cc:2229] T 252f4dfc0c284cd59b7361de730ab47f P d5bab5abc3724763bd421922c40d3b7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:34:43.505537 11220 raft_consensus.cc:2258] T 252f4dfc0c284cd59b7361de730ab47f P d5bab5abc3724763bd421922c40d3b7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:34:43.505797 11221 ts_tablet_manager.cc:1905] T 252f4dfc0c284cd59b7361de730ab47f P ee4ccf72494b4646b14245431b0badc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:43.507628 11220 ts_tablet_manager.cc:1905] T 252f4dfc0c284cd59b7361de730ab47f P d5bab5abc3724763bd421922c40d3b7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:43.514333 11220 ts_tablet_manager.cc:1918] T 252f4dfc0c284cd59b7361de730ab47f P d5bab5abc3724763bd421922c40d3b7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:34:43.514614 11220 log.cc:1196] T 252f4dfc0c284cd59b7361de730ab47f P d5bab5abc3724763bd421922c40d3b7e: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/252f4dfc0c284cd59b7361de730ab47f
I20260619 04:34:43.515187 11220 ts_tablet_manager.cc:1939] T 252f4dfc0c284cd59b7361de730ab47f P d5bab5abc3724763bd421922c40d3b7e: Deleting consensus metadata
I20260619 04:34:43.517140 10705 catalog_manager.cc:4841] TS d5bab5abc3724763bd421922c40d3b7e (127.0.97.3:42465): tablet 252f4dfc0c284cd59b7361de730ab47f (table test-table [id=2f71bc90186c4f399f7ffbb97e68cb67]) successfully deleted
I20260619 04:34:43.517622 11221 ts_tablet_manager.cc:1918] T 252f4dfc0c284cd59b7361de730ab47f P ee4ccf72494b4646b14245431b0badc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:34:43.517855 11221 log.cc:1196] T 252f4dfc0c284cd59b7361de730ab47f P ee4ccf72494b4646b14245431b0badc8: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-0/wal/wals/252f4dfc0c284cd59b7361de730ab47f
I20260619 04:34:43.518438 11221 ts_tablet_manager.cc:1939] T 252f4dfc0c284cd59b7361de730ab47f P ee4ccf72494b4646b14245431b0badc8: Deleting consensus metadata
I20260619 04:34:43.519829 11219 tablet_replica.cc:330] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a: stopping tablet replica
I20260619 04:34:43.520154 10705 catalog_manager.cc:4841] TS ee4ccf72494b4646b14245431b0badc8 (127.0.97.1:35495): tablet 252f4dfc0c284cd59b7361de730ab47f (table test-table [id=2f71bc90186c4f399f7ffbb97e68cb67]) successfully deleted
I20260619 04:34:43.520270 11219 raft_consensus.cc:2229] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:34:43.520895 11219 raft_consensus.cc:2258] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 19 04:34:43 dist-test-slave-n3tq krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 04:34:43.522876 11219 ts_tablet_manager.cc:1905] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:34:43.528088 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:43.494724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50640 (local address 127.0.97.1:35495)
0619 04:34:43.494902 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:43.494909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:43.512945 (+ 18036us) server_negotiation.cc:407] Connection header received
0619 04:34:43.513231 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:43.513237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:43.513318 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:43.513386 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:43.514977 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.516058 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.517891 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.518540 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.518635 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:43.522860 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:43.522871 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:43.522883 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:43.522912 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:43.525592 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:43.526190 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:43.526199 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:43.526203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:43.526329 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:43.526685 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:43.526690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:43.526694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:43.527025 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:43.527286 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:43.527549 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 04:34:43.527749 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:34:43.528736 10904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:43.530057 11219 ts_tablet_manager.cc:1918] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:34:43.530298 11219 log.cc:1196] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/252f4dfc0c284cd59b7361de730ab47f
I20260619 04:34:43.530933 11219 ts_tablet_manager.cc:1939] T 252f4dfc0c284cd59b7361de730ab47f P 0623a426159d463fbf10bea7ff042b0a: Deleting consensus metadata
I20260619 04:34:43.532827 10704 catalog_manager.cc:4841] TS 0623a426159d463fbf10bea7ff042b0a (127.0.97.2:35883): tablet 252f4dfc0c284cd59b7361de730ab47f (table test-table [id=2f71bc90186c4f399f7ffbb97e68cb67]) successfully deleted
I20260619 04:34:43.614789 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 10781
I20260619 04:34:43.629091 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 10915
Jun 19 04:34:43 dist-test-slave-n3tq krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 04:34:43.644671 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:43.629844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35846 (local address 127.0.97.2:35883)
0619 04:34:43.630046 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:43.630052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:43.630072 (+ 20us) server_negotiation.cc:407] Connection header received
0619 04:34:43.630193 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:43.630198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:43.630280 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:43.630371 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:43.631462 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.632689 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.633913 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.634580 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.634701 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:43.639013 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:43.639030 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:43.639046 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:43.639088 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:43.642058 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:43.642827 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:43.642833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:43.642836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:43.642983 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:43.643338 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:43.643341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:43.643343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:43.643657 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:43.643886 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:43.644151 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 04:34:43.644312 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 04:34:43.645354 11038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:43.756601 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 10915
I20260619 04:34:43.770812 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 11048
Jun 19 04:34:43 dist-test-slave-n3tq krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 04:34:43.784654 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:34:43.771537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35576 (local address 127.0.97.3:42465)
0619 04:34:43.771705 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 04:34:43.771710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:34:43.771730 (+ 20us) server_negotiation.cc:407] Connection header received
0619 04:34:43.771889 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:34:43.771893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:34:43.771953 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:34:43.772034 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:34:43.773340 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.774030 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.775215 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:34:43.775608 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:34:43.775713 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:34:43.779679 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:34:43.779692 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:34:43.779709 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:34:43.779744 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:34:43.782157 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:43.782743 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:43.782750 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:43.782753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:43.782882 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:34:43.783213 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:34:43.783217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:34:43.783219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:34:43.783552 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:34:43.783809 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 04:34:43.784137 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 04:34:43.784344 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 04:34:43.785248 11171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:43.870697 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 11048
I20260619 04:34:43.883406 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 10688
I20260619 04:34:43.884435 10750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:34:43.988555 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 10688
2026-06-19T04:34:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3947 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:34:44 dist-test-slave-n3tq krb5kdc[11229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:34:44 dist-test-slave-n3tq krb5kdc[11229](info): set up 2 sockets
Jun 19 04:34:44 dist-test-slave-n3tq krb5kdc[11229](info): commencing operation
krb5kdc: starting...
W20260619 04:34:44.989292 388 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:34:45 dist-test-slave-n3tq krb5kdc[11229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:34:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:34:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260619 04:34:45.277801 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task24iQRF/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/postgres -l logfile start
2026-06-19 04:34:47.522 UTC [11259] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 04:34:47.523 UTC [11259] LOG: listening on IPv4 address "127.0.97.20", port 38747
2026-06-19 04:34:47.525 UTC [11259] LOG: listening on Unix socket "/tmp/.s.PGSQL.38747"
2026-06-19 04:34:47.545 UTC [11262] LOG: database system was shut down at 2026-06-19 04:34:46 UTC
2026-06-19 04:34:47.549 UTC [11259] LOG: database system is ready to accept connections
I20260619 04:34:48.436728 388 mini_postgres.cc:96] Postgres bound to 38747
2026-06-19 04:34:48.452 UTC [11270] FATAL: database "slave" does not exist
127.0.97.20:38747 - accepting connections
I20260619 04:34:48.453142 388 mini_ranger.cc:161] Starting Ranger...
I20260619 04:34:48.476217 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 04:34:48.867635 388 mini_ranger.cc:87] Created ranger Postgres database
I20260619 04:34:48.867790 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ranger-admin
2026-06-19 04:34:49,133 [I] DB FLAVOR :POSTGRES
2026-06-19 04:34:49,133 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 04:34:49,133 [I] Checking connection..
2026-06-19 04:34:49,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 04:34:49,377 [I] Checking connection passed.
2026-06-19 04:34:49,377 [I] --------- Verifying version history table ---------
2026-06-19 04:34:49,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 04:34:49,612 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 04:34:49,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 04:34:49,883 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 04:34:49,884 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 04:34:49,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 04:34:50.113 UTC [11334] WARNING: there is no transaction in progress
2026-06-19 04:34:50,119 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 04:34:50,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 04:34:50,353 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 04:34:50,354 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 04:34:50,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 04:34:50,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 04:34:50,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-n3tq.c.gcp-upstream.internal','N') ;"
2026-06-19 04:34:51,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 04:34:51,277 [I] Table x_portal_user does not exist in database ranger
2026-06-19 04:34:51,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 04:34:51,526 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 04:34:51,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 04:34:51,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 04:34:52,000 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 04:34:52,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 04:34:52.870 UTC [11442] WARNING: there is no transaction in progress
2026-06-19 04:34:52.885 UTC [11442] WARNING: there is no transaction in progress
2026-06-19 04:34:52.899 UTC [11442] WARNING: there is no transaction in progress
2026-06-19 04:34:52.912 UTC [11442] WARNING: there is no transaction in progress
2026-06-19 04:34:52.926 UTC [11442] WARNING: there is no transaction in progress
2026-06-19 04:34:52.942 UTC [11442] WARNING: there is no transaction in progress
2026-06-19 04:34:52.991 UTC [11442] WARNING: there is no transaction in progress
2026-06-19 04:34:52.997 UTC [11442] WARNING: there is no transaction in progress
2026-06-19 04:34:53.009 UTC [11442] WARNING: there is no transaction in progress
2026-06-19 04:34:53.018 UTC [11442] WARNING: there is no transaction in progress
2026-06-19 04:34:53.028 UTC [11442] WARNING: there is no transaction in progress
2026-06-19 04:34:53.040 UTC [11442] WARNING: there is no transaction in progress
2026-06-19 04:34:53.049 UTC [11442] WARNING: there is no transaction in progress
2026-06-19 04:34:53.054 UTC [11442] WARNING: there is no transaction in progress
2026-06-19 04:34:53.060 UTC [11442] WARNING: there is no transaction in progress
2026-06-19 04:34:53.065 UTC [11442] WARNING: there is no transaction in progress
2026-06-19 04:34:53,606 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 04:34:53,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 04:34:53,838 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 04:34:53,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 04:34:54,087 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 04:34:54,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 04:34:54,338 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 04:34:54,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 04:34:54,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 04:34:54,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-n3tq.c.gcp-upstream.internal';"
2026-06-19 04:34:55,055 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 04:34:55,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 04:34:55,299 [I] DB_PATCHES have already been applied
I20260619 04:34:55.304891 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ranger-admin:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task24iQRF/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 04:34:55.305009 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260619 04:34:55.318154 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:42757
Jun 19, 2026 4:34:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:34:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:34:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 4:34:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 4:34:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 4:34:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 4:34:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 04:34:56.347550 388 mini_ranger.cc:160] Time spent starting Ranger: real 7.894s user 0.008s sys 0.028s
Jun 19, 2026 4:35:15 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 4:35:15 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 4:35:15 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 4:35:15 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 4:35:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 4:35:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 4:35:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 4:35:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 4:35:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 4:35:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 4:35:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 4:35:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 4:35:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 4:35:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 4:35:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 4:35:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 4:35:15 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 4:35:16 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 04:35:17.394093 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 04:35:17 dist-test-slave-n3tq krb5kdc[11229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843717, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 04:35:17.562894 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 04:35:17.584304 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 04:35:18.309466 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 04:35:18.309597 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ranger-kms
I20260619 04:35:18.309739 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:38747
2026-06-19 04:35:18,702 [I] DB FLAVOR :POSTGRES
2026-06-19 04:35:18,702 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 04:35:18,702 [I] Checking connection
2026-06-19 04:35:18,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 04:35:18,933 [I] connection success
2026-06-19 04:35:18,933 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 04:35:18,933 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 04:35:18,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 04:35:19,155 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 04:35:19,156 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 04:35:19,156 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 04:35:19,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38747/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 04:35:19,402 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 04:35:19.822222 388 mini_ranger_kms.cc:310] Created kms service
I20260619 04:35:20.009478 388 mini_ranger_kms.cc:326] Created kudu user
I20260619 04:35:20.052268 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 04:35:20 dist-test-slave-n3tq krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843717, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 04:35:20.427380 388 mini_ranger_kms.cc:384] Added ranger policy
I20260619 04:35:20.427645 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ranger-kms:/tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 04:35:20.427699 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260619 04:35:20.447736 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11622
I20260619 04:35:20.447893 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:56377
Jun 19, 2026 4:35:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:35:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:35:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 4:35:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 4:35:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 4:35:21 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 04:35:22 dist-test-slave-n3tq krb5kdc[11229](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781843722, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:35:22.506258 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.943s user 0.116s sys 0.298s
I20260619 04:35:22.506475 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 04:35:22.506655 388 mini_ranger_kms.cc:401] 127.0.97.20:56377/kms/v1/keys
Jun 19 04:35:22 dist-test-slave-n3tq krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843722, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 04:35:23 dist-test-slave-n3tq krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843722, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 04:35:25 dist-test-slave-n3tq krb5kdc[11229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:35:25.563700 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35503
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35503
--ranger_config_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56377/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:35:25.900516 11673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:35:25.900794 11673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:35:25.900884 11673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:35:25.907955 11673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:35:25.908051 11673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:35:25.908103 11673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:35:25.908154 11673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:35:25.908200 11673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:35:25.918350 11673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:56377/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35503
--ranger_config_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:35:25.919534 11673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:35:25.920769 11673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:35:25.931931 11678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:35:25.931879 11679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:35:25.932300 11681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:35:25.932716 11673 server_base.cc:1034] running on GCE node
I20260619 04:35:25.933458 11673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:35:25.934795 11673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:35:25.936015 11673 hybrid_clock.cc:648] HybridClock initialized: now 1781843725935963 us; error 47 us; skew 500 ppm
I20260619 04:35:25.936371 11673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:35:25 dist-test-slave-n3tq krb5kdc[11229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:35:25.939411 11673 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:35:25.940888 11673 webserver.cc:460] Webserver started at http://127.0.97.62:34091/ using document root <none> and password file <none>
I20260619 04:35:25.941515 11673 fs_manager.cc:359] Metadata directory not provided
I20260619 04:35:25.941612 11673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:35:25.941900 11673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 04:35:25 dist-test-slave-n3tq krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 04:35:25.990008 11673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "889fc7d1dd844080847c809ec6e5509c"
format_stamp: "Formatted at 2026-06-19 04:35:25 on dist-test-slave-n3tq"
server_key: "f8610a9c63a24ce00f99b92461c81252"
server_key_iv: "1f373edfeb7345538dc43dd389607f60"
server_key_version: "kuduclusterkey@0"
I20260619 04:35:25.990815 11673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "889fc7d1dd844080847c809ec6e5509c"
format_stamp: "Formatted at 2026-06-19 04:35:25 on dist-test-slave-n3tq"
server_key: "f8610a9c63a24ce00f99b92461c81252"
server_key_iv: "1f373edfeb7345538dc43dd389607f60"
server_key_version: "kuduclusterkey@0"
I20260619 04:35:25.996376 11673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 04:35:26.016443 11688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:35:26.017967 11673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:35:26.018129 11673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "889fc7d1dd844080847c809ec6e5509c"
format_stamp: "Formatted at 2026-06-19 04:35:25 on dist-test-slave-n3tq"
server_key: "f8610a9c63a24ce00f99b92461c81252"
server_key_iv: "1f373edfeb7345538dc43dd389607f60"
server_key_version: "kuduclusterkey@0"
I20260619 04:35:26.018335 11673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:35:26.033800 11673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:35:26.037586 11673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:35:26.037871 11673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:35:26.057022 11673 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35503
I20260619 04:35:26.057039 11739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35503 every 8 connection(s)
I20260619 04:35:26.058338 11673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:35:26.062250 11740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:35:26.067411 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 11673
I20260619 04:35:26.067610 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:35:26.068379 388 external_mini_cluster.cc:1428] Setting key d24b20b6498866ca25b3930e4be23878
I20260619 04:35:26.074977 11740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c: Bootstrap starting.
I20260619 04:35:26.079403 11740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c: Neither blocks nor log segments found. Creating new log.
I20260619 04:35:26.080928 11740 log.cc:824] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c: Log is configured to *not* fsync() on all Append() calls
I20260619 04:35:26.087566 11740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c: No bootstrap required, opened a new log
Jun 19 04:35:26 dist-test-slave-n3tq krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:35:26.096799 11740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889fc7d1dd844080847c809ec6e5509c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35503 } }
I20260619 04:35:26.097256 11740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:35:26.097360 11740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889fc7d1dd844080847c809ec6e5509c, State: Initialized, Role: FOLLOWER
I20260619 04:35:26.097988 11740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889fc7d1dd844080847c809ec6e5509c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35503 } }
I20260619 04:35:26.098222 11740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:35:26.098356 11740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:35:26.098503 11740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:35:26.100756 11740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889fc7d1dd844080847c809ec6e5509c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35503 } }
I20260619 04:35:26.101298 11740 leader_election.cc:302] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 889fc7d1dd844080847c809ec6e5509c; no voters:
I20260619 04:35:26.102217 11740 leader_election.cc:288] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:35:26.102322 11745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:35:26.102872 11745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c [term 1 LEADER]: Becoming Leader. State: Replica: 889fc7d1dd844080847c809ec6e5509c, State: Running, Role: LEADER
I20260619 04:35:26.103494 11745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889fc7d1dd844080847c809ec6e5509c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35503 } }
I20260619 04:35:26.106079 11740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:35:26.106479 11743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:35:26.070578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57266 (local address 127.0.97.62:35503)
0619 04:35:26.084979 (+ 14401us) server_negotiation.cc:206] Beginning negotiation
0619 04:35:26.084995 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 04:35:26.085054 (+ 59us) server_negotiation.cc:407] Connection header received
0619 04:35:26.087237 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:35:26.087262 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:35:26.087693 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:35:26.088055 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:35:26.089617 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:26.090917 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:26.092665 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:26.093355 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:26.093490 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:35:26.097860 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:35:26.097868 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:35:26.097883 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:35:26.097915 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:35:26.101197 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:35:26.102026 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:35:26.102033 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:35:26.102043 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:35:26.102181 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:35:26.102565 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:35:26.102569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:35:26.102571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:35:26.103350 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:35:26.103658 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 04:35:26.104632 (+ 974us) server_negotiation.cc:299] Negotiation successful
0619 04:35:26.105027 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14163,"thread_start_us":13944,"threads_started":1}
I20260619 04:35:26.106244 11747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "889fc7d1dd844080847c809ec6e5509c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889fc7d1dd844080847c809ec6e5509c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35503 } } }
I20260619 04:35:26.106434 11746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 889fc7d1dd844080847c809ec6e5509c. Latest consensus state: current_term: 1 leader_uuid: "889fc7d1dd844080847c809ec6e5509c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889fc7d1dd844080847c809ec6e5509c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35503 } } }
I20260619 04:35:26.106865 11746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c [sys.catalog]: This master's current role is: LEADER
I20260619 04:35:26.106869 11747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c [sys.catalog]: This master's current role is: LEADER
I20260619 04:35:26.107741 11749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:35:26.117283 11749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:35:26.125643 11740 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 04:35:26.126523 11749 catalog_manager.cc:1347] Generated new cluster ID: 1f068f27dfe446d6ba81ae386f4dffad
I20260619 04:35:26.126612 11749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:35:26.153070 11749 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:35:26.155094 11749 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:35:26.163815 11749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c: Generated new TSK 0
I20260619 04:35:26.164891 11749 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 04:35:26.875689 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56377/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35503
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:35:27 dist-test-slave-n3tq krb5kdc[11229](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781843727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 04:35:27.270638 11785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:35:27.270938 11785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:35:27.271095 11785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:35:27.278493 11785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:35:27.278623 11785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:35:27.278826 11785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:35:27.291052 11785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:56377/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:35:27.292591 11785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:35:27.294044 11785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:35:27.310385 11796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:35:27.310884 11795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:35:27.318723 11799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:35:27.318974 11785 server_base.cc:1034] running on GCE node
I20260619 04:35:27.319868 11785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:35:27.321170 11785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:35:27.322469 11785 hybrid_clock.cc:648] HybridClock initialized: now 1781843727322389 us; error 71 us; skew 500 ppm
I20260619 04:35:27.322937 11785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:35:27 dist-test-slave-n3tq krb5kdc[11229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:35:27.327024 11785 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:35:27.329042 11785 webserver.cc:460] Webserver started at http://127.0.97.1:42819/ using document root <none> and password file <none>
I20260619 04:35:27.329985 11785 fs_manager.cc:359] Metadata directory not provided
I20260619 04:35:27.330129 11785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:35:27.330524 11785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 04:35:27 dist-test-slave-n3tq krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 04:35:27.350554 11785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "010078eddda04c98a4894df9d73fd630"
format_stamp: "Formatted at 2026-06-19 04:35:27 on dist-test-slave-n3tq"
server_key: "d5245fb133fcabff988d19821c6864c5"
server_key_iv: "ed4ebcde4cc922602863f7405c393d16"
server_key_version: "kuduclusterkey@0"
I20260619 04:35:27.351788 11785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "010078eddda04c98a4894df9d73fd630"
format_stamp: "Formatted at 2026-06-19 04:35:27 on dist-test-slave-n3tq"
server_key: "d5245fb133fcabff988d19821c6864c5"
server_key_iv: "ed4ebcde4cc922602863f7405c393d16"
server_key_version: "kuduclusterkey@0"
I20260619 04:35:27.360868 11785 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20260619 04:35:27.400143 11805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:35:27.402201 11785 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.006s sys 0.000s
I20260619 04:35:27.402433 11785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "010078eddda04c98a4894df9d73fd630"
format_stamp: "Formatted at 2026-06-19 04:35:27 on dist-test-slave-n3tq"
server_key: "d5245fb133fcabff988d19821c6864c5"
server_key_iv: "ed4ebcde4cc922602863f7405c393d16"
server_key_version: "kuduclusterkey@0"
I20260619 04:35:27.402688 11785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:35:27.454319 11785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:35:27.459607 11785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:35:27.460013 11785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:35:27.467428 11785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:35:27.474860 11785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:35:27.475018 11785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 04:35:27.475178 11785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:35:27.475256 11785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 04:35:27 dist-test-slave-n3tq krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 04:35:27.967581 11917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40213 every 8 connection(s)
I20260619 04:35:27.967931 11785 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40213
I20260619 04:35:27.969753 11785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:35:27.974335 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 11785
I20260619 04:35:27.974540 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:35:27.975308 388 external_mini_cluster.cc:1428] Setting key ff0e759b19d681d5b2a733a836424eef
Jun 19 04:35:27 dist-test-slave-n3tq krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:35:28.009544 11921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:35:27.978014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35503 (local address 127.0.97.1:52007)
0619 04:35:27.989485 (+ 11471us) negotiation.cc:107] Waiting for socket to connect
0619 04:35:27.989541 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 04:35:27.991879 (+ 2338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:35:27.992677 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:35:27.992687 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:35:27.993150 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:35:27.993762 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:35:27.993781 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:35:27.994840 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:35:27.994845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:35:27.996077 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:35:27.996100 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:35:27.996916 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:35:27.996924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:35:27.997145 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:35:27.997831 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:35:27.997863 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:35:28.000557 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:35:28.004536 (+ 3979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:35:28.004546 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:35:28.004566 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:35:28.005154 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:35:28.005596 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:35:28.005601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:35:28.005605 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:35:28.005883 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:35:28.006634 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:35:28.006642 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:35:28.007099 (+ 457us) client_negotiation.cc:769] Sending connection context
0619 04:35:28.007448 (+ 349us) client_negotiation.cc:240] Negotiation successful
0619 04:35:28.008013 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11103,"thread_start_us":11383,"threads_started":1}
I20260619 04:35:28.010838 11920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:35:27.977171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52007 (local address 127.0.97.62:35503)
0619 04:35:27.989498 (+ 12327us) server_negotiation.cc:206] Beginning negotiation
0619 04:35:27.989505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:35:27.990360 (+ 855us) server_negotiation.cc:407] Connection header received
0619 04:35:27.992172 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:35:27.992177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:35:27.992276 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:35:27.992372 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:35:27.993937 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:27.994681 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:27.996241 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:27.996691 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:27.996805 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:35:28.000863 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:35:28.000870 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:35:28.000875 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:35:28.000916 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:35:28.004337 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:35:28.005316 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:35:28.005321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:35:28.005325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:35:28.005466 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:35:28.006038 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:35:28.006043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:35:28.006047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:35:28.006396 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:35:28.006550 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 04:35:28.010128 (+ 3578us) server_negotiation.cc:299] Negotiation successful
0619 04:35:28.010401 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12210,"thread_start_us":12094,"threads_started":1}
I20260619 04:35:28.013221 11918 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35503
I20260619 04:35:28.013590 11918 heartbeater.cc:464] Registering TS with master...
I20260619 04:35:28.014621 11918 heartbeater.cc:511] Master 127.0.97.62:35503 requested a full tablet report, sending...
I20260619 04:35:28.017259 11705 ts_manager.cc:194] Registered new tserver with Master: 010078eddda04c98a4894df9d73fd630 (127.0.97.1:40213)
I20260619 04:35:28.019207 11705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52007
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 04:35:28.061492 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56377/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35503
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:35:28 dist-test-slave-n3tq krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260619 04:35:28.525910 11924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:35:28.526276 11924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:35:28.526415 11924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:35:28.537089 11924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:35:28.537223 11924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:35:28.537387 11924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:35:28.552153 11924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:56377/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:35:28.553843 11924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:35:28.555606 11924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:35:28.572273 11938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:35:28.572328 11941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:35:28.572293 11939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:35:28.573683 11924 server_base.cc:1034] running on GCE node
I20260619 04:35:28.574381 11924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:35:28.575423 11924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:35:28.576632 11924 hybrid_clock.cc:648] HybridClock initialized: now 1781843728576575 us; error 40 us; skew 500 ppm
I20260619 04:35:28.576979 11924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:35:28 dist-test-slave-n3tq krb5kdc[11229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:35:28.580320 11924 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:35:28.581904 11924 webserver.cc:460] Webserver started at http://127.0.97.2:40107/ using document root <none> and password file <none>
I20260619 04:35:28.582580 11924 fs_manager.cc:359] Metadata directory not provided
I20260619 04:35:28.582692 11924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:35:28.582989 11924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 04:35:28 dist-test-slave-n3tq krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 04:35:28.599753 11924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "659be81aeb1d446988584cb79dd4f0fd"
format_stamp: "Formatted at 2026-06-19 04:35:28 on dist-test-slave-n3tq"
server_key: "c607464276a232fc941415360d86b663"
server_key_iv: "c17674d3516c08984cbac96a8f383582"
server_key_version: "kuduclusterkey@0"
I20260619 04:35:28.600930 11924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "659be81aeb1d446988584cb79dd4f0fd"
format_stamp: "Formatted at 2026-06-19 04:35:28 on dist-test-slave-n3tq"
server_key: "c607464276a232fc941415360d86b663"
server_key_iv: "c17674d3516c08984cbac96a8f383582"
server_key_version: "kuduclusterkey@0"
I20260619 04:35:28.609666 11924 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260619 04:35:28.628808 11948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:35:28.630683 11924 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 04:35:28.630923 11924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "659be81aeb1d446988584cb79dd4f0fd"
format_stamp: "Formatted at 2026-06-19 04:35:28 on dist-test-slave-n3tq"
server_key: "c607464276a232fc941415360d86b663"
server_key_iv: "c17674d3516c08984cbac96a8f383582"
server_key_version: "kuduclusterkey@0"
I20260619 04:35:28.631242 11924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:35:28.659727 11924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:35:28.664829 11924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:35:28.665138 11924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:35:28.666250 11924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:35:28.667897 11924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:35:28.667994 11924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:35:28.668114 11924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:35:28.668175 11924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:35:28.700541 11924 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34889
I20260619 04:35:28.700562 12060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34889 every 8 connection(s)
I20260619 04:35:28.702193 11924 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:35:28.705976 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 11924
I20260619 04:35:28.706157 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:35:28.706650 388 external_mini_cluster.cc:1428] Setting key ec2d6c685c8818d6be3e3f1c27ac9c49
Jun 19 04:35:28 dist-test-slave-n3tq krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:35:28.727696 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:35:28.704810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59785 (local address 127.0.97.62:35503)
0619 04:35:28.705230 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 04:35:28.705234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:35:28.707343 (+ 2109us) server_negotiation.cc:407] Connection header received
0619 04:35:28.709712 (+ 2369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:35:28.709718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:35:28.709813 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:35:28.709917 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:35:28.711757 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:28.712775 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:28.714993 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:28.715520 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:28.715642 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:35:28.719629 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:35:28.719636 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:35:28.719640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:35:28.719677 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:35:28.722985 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:35:28.723955 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:35:28.723961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:35:28.723964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:35:28.724094 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:35:28.724618 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:35:28.724622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:35:28.724625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:35:28.724973 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:35:28.725097 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 04:35:28.727080 (+ 1983us) server_negotiation.cc:299] Negotiation successful
0619 04:35:28.727300 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":229,"threads_started":1}
I20260619 04:35:28.727891 12064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:35:28.705483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35503 (local address 127.0.97.2:59785)
0619 04:35:28.706633 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0619 04:35:28.706695 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 04:35:28.709396 (+ 2701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:35:28.710192 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:35:28.710204 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:35:28.710754 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:35:28.711549 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:35:28.711575 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:35:28.712957 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:35:28.712963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:35:28.714851 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:35:28.714866 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:35:28.715743 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:35:28.715750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:35:28.716021 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:35:28.716792 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:35:28.716824 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:35:28.719379 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:35:28.723152 (+ 3773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:35:28.723162 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:35:28.723247 (+ 85us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:35:28.723818 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:35:28.724205 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:35:28.724209 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:35:28.724213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:35:28.724494 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:35:28.725161 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:35:28.725169 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:35:28.725609 (+ 440us) client_negotiation.cc:769] Sending connection context
0619 04:35:28.725909 (+ 300us) client_negotiation.cc:240] Negotiation successful
0619 04:35:28.726440 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":333,"threads_started":1}
I20260619 04:35:28.729599 12061 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35503
I20260619 04:35:28.729926 12061 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 04:35:28.731241 12061 heartbeater.cc:511] Master 127.0.97.62:35503 requested a full tablet report, sending...
I20260619 04:35:28.732963 11705 ts_manager.cc:194] Registered new tserver with Master: 659be81aeb1d446988584cb79dd4f0fd (127.0.97.2:34889)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 04:35:28.734130 11705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 04:35:28.757426 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56377/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35503
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:35:29.023072 11918 heartbeater.cc:503] Master 127.0.97.62:35503 was elected leader, sending a full tablet report...
W20260619 04:35:29.092183 12067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:35:29.092504 12067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:35:29.092639 12067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:35:29.099876 12067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:35:29.100014 12067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:35:29.100163 12067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:35:29.110697 12067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:56377/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:35:29.112022 12067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:35:29.113354 12067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:35:29.126258 12072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:35:29.126425 12075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:35:29.126274 12073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:35:29.127533 12067 server_base.cc:1034] running on GCE node
I20260619 04:35:29.128137 12067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:35:29.129153 12067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:35:29.130344 12067 hybrid_clock.cc:648] HybridClock initialized: now 1781843729130292 us; error 49 us; skew 500 ppm
I20260619 04:35:29.130645 12067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:35:29 dist-test-slave-n3tq krb5kdc[11229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:35:29.133668 12067 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:35:29.135098 12067 webserver.cc:460] Webserver started at http://127.0.97.3:40187/ using document root <none> and password file <none>
I20260619 04:35:29.135761 12067 fs_manager.cc:359] Metadata directory not provided
I20260619 04:35:29.135866 12067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:35:29.136157 12067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 04:35:29 dist-test-slave-n3tq krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 04:35:29.150236 12067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "8ef28c8d03ca4f46acee4b5a97e6e9e9"
format_stamp: "Formatted at 2026-06-19 04:35:29 on dist-test-slave-n3tq"
server_key: "a10bd6985a0b40d7c69f92cc9832d0f6"
server_key_iv: "f38e0e5f9ac631ac21df41f74e4cec30"
server_key_version: "kuduclusterkey@0"
I20260619 04:35:29.151055 12067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8ef28c8d03ca4f46acee4b5a97e6e9e9"
format_stamp: "Formatted at 2026-06-19 04:35:29 on dist-test-slave-n3tq"
server_key: "a10bd6985a0b40d7c69f92cc9832d0f6"
server_key_iv: "f38e0e5f9ac631ac21df41f74e4cec30"
server_key_version: "kuduclusterkey@0"
I20260619 04:35:29.156411 12067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 04:35:29.173749 12082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:35:29.175372 12067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:35:29.175539 12067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "8ef28c8d03ca4f46acee4b5a97e6e9e9"
format_stamp: "Formatted at 2026-06-19 04:35:29 on dist-test-slave-n3tq"
server_key: "a10bd6985a0b40d7c69f92cc9832d0f6"
server_key_iv: "f38e0e5f9ac631ac21df41f74e4cec30"
server_key_version: "kuduclusterkey@0"
I20260619 04:35:29.175781 12067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:35:29.187943 12067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:35:29.191692 12067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:35:29.191952 12067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:35:29.192905 12067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:35:29.194221 12067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:35:29.194305 12067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:35:29.194424 12067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:35:29.194469 12067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:35:29.228510 12067 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33821
I20260619 04:35:29.228533 12194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33821 every 8 connection(s)
I20260619 04:35:29.229763 12067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:35:29.239156 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 12067
I20260619 04:35:29.239331 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:35:29.239822 388 external_mini_cluster.cc:1428] Setting key 8b21fcb270216afdecb5b8e6b218fadc
Jun 19 04:35:29 dist-test-slave-n3tq krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:35:29.249260 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:35:29.232148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57243 (local address 127.0.97.62:35503)
0619 04:35:29.232581 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0619 04:35:29.232586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:35:29.233396 (+ 810us) server_negotiation.cc:407] Connection header received
0619 04:35:29.235220 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:35:29.235225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:35:29.235318 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:35:29.235414 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:35:29.236917 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.237643 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.239452 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.239866 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.239996 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:35:29.243714 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:35:29.243720 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:35:29.243725 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:35:29.243756 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:35:29.246306 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:35:29.247064 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:35:29.247069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:35:29.247071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:35:29.247191 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:35:29.247614 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:35:29.247617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:35:29.247619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:35:29.247882 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:35:29.248007 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 04:35:29.248702 (+ 695us) server_negotiation.cc:299] Negotiation successful
0619 04:35:29.248912 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":226,"threads_started":1}
I20260619 04:35:29.250196 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:35:29.232581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35503 (local address 127.0.97.3:57243)
0619 04:35:29.233239 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0619 04:35:29.233281 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 04:35:29.234955 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:35:29.235658 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:35:29.235666 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:35:29.236074 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:35:29.236756 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:35:29.236775 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.237791 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.237795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:35:29.239306 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:35:29.239320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.240079 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.240087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:35:29.240309 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:35:29.240995 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:35:29.241025 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:35:29.243497 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:35:29.246465 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:35:29.246475 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:35:29.246494 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:35:29.246941 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:35:29.247292 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:35:29.247297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:35:29.247301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:35:29.247504 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:35:29.247998 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:35:29.248006 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:35:29.248332 (+ 326us) client_negotiation.cc:769] Sending connection context
0619 04:35:29.248598 (+ 266us) client_negotiation.cc:240] Negotiation successful
0619 04:35:29.249133 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":260,"threads_started":1}
I20260619 04:35:29.251461 12195 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35503
I20260619 04:35:29.251727 12195 heartbeater.cc:464] Registering TS with master...
I20260619 04:35:29.252405 12195 heartbeater.cc:511] Master 127.0.97.62:35503 requested a full tablet report, sending...
I20260619 04:35:29.253897 11704 ts_manager.cc:194] Registered new tserver with Master: 8ef28c8d03ca4f46acee4b5a97e6e9e9 (127.0.97.3:33821)
I20260619 04:35:29.255100 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:35:29.255125 11704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57243
I20260619 04:35:29.272359 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:35:29.260182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43066 (local address 127.0.97.62:35503)
0619 04:35:29.260327 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 04:35:29.260331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:35:29.260731 (+ 400us) server_negotiation.cc:407] Connection header received
0619 04:35:29.260871 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:35:29.260875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:35:29.260945 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:35:29.261028 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:35:29.262293 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.262979 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.264788 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.265218 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.265341 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:35:29.267639 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:35:29.267645 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:35:29.267648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:35:29.267690 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:35:29.270038 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:35:29.270653 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:35:29.270656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:35:29.270659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:35:29.270780 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:35:29.271205 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:35:29.271208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:35:29.271211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:35:29.271492 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:35:29.271664 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 04:35:29.271970 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 04:35:29.272138 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 04:35:29.276659 11704 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:35:29.283775 11704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:35:29.320560 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:35:29.296984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37414 (local address 127.0.97.1:40213)
0619 04:35:29.310900 (+ 13916us) server_negotiation.cc:206] Beginning negotiation
0619 04:35:29.310909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:35:29.310934 (+ 25us) server_negotiation.cc:407] Connection header received
0619 04:35:29.311016 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:35:29.311022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:35:29.311227 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:35:29.311410 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:35:29.312649 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.314468 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.317522 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.319000 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.319114 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:35:29.319333 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 04:35:29.320003 (+ 670us) server_negotiation.cc:299] Negotiation successful
0619 04:35:29.320171 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13750,"thread_start_us":14730,"threads_started":1}
I20260619 04:35:29.320791 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:35:29.297530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57850 (local address 127.0.97.3:33821)
0619 04:35:29.310608 (+ 13078us) server_negotiation.cc:206] Beginning negotiation
0619 04:35:29.310618 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:35:29.310653 (+ 35us) server_negotiation.cc:407] Connection header received
0619 04:35:29.310768 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:35:29.310776 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:35:29.311062 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:35:29.311266 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:35:29.312406 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.314468 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.317560 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.318772 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.318887 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:35:29.319122 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 04:35:29.320146 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0619 04:35:29.320382 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12883,"thread_start_us":13952,"threads_started":1}
I20260619 04:35:29.321408 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:35:29.296904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33821 (local address 127.0.0.1:57850)
0619 04:35:29.308520 (+ 11616us) negotiation.cc:107] Waiting for socket to connect
0619 04:35:29.308575 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 04:35:29.309626 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:35:29.311657 (+ 2031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:35:29.311664 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:35:29.311694 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:35:29.312135 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:35:29.312148 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.314622 (+ 2474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.314627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:35:29.317362 (+ 2735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:35:29.317376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.319462 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.319472 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:35:29.319832 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:35:29.319863 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 04:35:29.319944 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 04:35:29.320992 (+ 1048us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11267,"mutex_wait_us":158,"thread_start_us":11079,"threads_started":1}
I20260619 04:35:29.321450 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:35:29.296952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40213 (local address 127.0.0.1:37414)
0619 04:35:29.310574 (+ 13622us) negotiation.cc:107] Waiting for socket to connect
0619 04:35:29.310589 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:35:29.310678 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:35:29.311936 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:35:29.311940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:35:29.311961 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:35:29.312400 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:35:29.312411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.314711 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.314715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:35:29.317374 (+ 2659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:35:29.317383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.319496 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.319503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:35:29.319715 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:35:29.319773 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 04:35:29.319928 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 04:35:29.321014 (+ 1086us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13531,"mutex_wait_us":906,"thread_start_us":13781,"threads_started":1}
I20260619 04:35:29.324976 12130 tablet_service.cc:1467] Processing CreateTablet for tablet 99c8ee2140b747a3874f8fbd116bfd60 (DEFAULT_TABLE table=test-table [id=4350c218d5534f058f6f00e26d97afba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:35:29.324978 11851 tablet_service.cc:1467] Processing CreateTablet for tablet 99c8ee2140b747a3874f8fbd116bfd60 (DEFAULT_TABLE table=test-table [id=4350c218d5534f058f6f00e26d97afba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:35:29.326242 11851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c8ee2140b747a3874f8fbd116bfd60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:35:29.326277 12130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c8ee2140b747a3874f8fbd116bfd60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:35:29.332345 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:35:29.311051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34889 (local address 127.0.0.1:40046)
0619 04:35:29.315698 (+ 4647us) negotiation.cc:107] Waiting for socket to connect
0619 04:35:29.315712 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 04:35:29.315826 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:35:29.316339 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:35:29.316343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:35:29.316364 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:35:29.316787 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:35:29.316796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.318422 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.318427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:35:29.321090 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:35:29.321105 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.331413 (+ 10308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.331426 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:35:29.331630 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:35:29.331668 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 04:35:29.331786 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 04:35:29.331942 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4528,"mutex_wait_us":1,"thread_start_us":4381,"threads_started":1}
I20260619 04:35:29.332424 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:35:29.311305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40046 (local address 127.0.97.2:34889)
0619 04:35:29.315295 (+ 3990us) server_negotiation.cc:206] Beginning negotiation
0619 04:35:29.315304 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:35:29.315820 (+ 516us) server_negotiation.cc:407] Connection header received
0619 04:35:29.315954 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:35:29.315959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:35:29.316167 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:35:29.316327 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:35:29.316967 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.318265 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.321257 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.322437 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.322556 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:35:29.322754 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 04:35:29.331917 (+ 9163us) server_negotiation.cc:299] Negotiation successful
0619 04:35:29.332112 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3814,"thread_start_us":3718,"threads_started":1}
I20260619 04:35:29.336668 11996 tablet_service.cc:1467] Processing CreateTablet for tablet 99c8ee2140b747a3874f8fbd116bfd60 (DEFAULT_TABLE table=test-table [id=4350c218d5534f058f6f00e26d97afba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:35:29.338014 12214 tablet_bootstrap.cc:492] T 99c8ee2140b747a3874f8fbd116bfd60 P 010078eddda04c98a4894df9d73fd630: Bootstrap starting.
I20260619 04:35:29.338117 11996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c8ee2140b747a3874f8fbd116bfd60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:35:29.341111 12215 tablet_bootstrap.cc:492] T 99c8ee2140b747a3874f8fbd116bfd60 P 8ef28c8d03ca4f46acee4b5a97e6e9e9: Bootstrap starting.
I20260619 04:35:29.341948 12214 tablet_bootstrap.cc:654] T 99c8ee2140b747a3874f8fbd116bfd60 P 010078eddda04c98a4894df9d73fd630: Neither blocks nor log segments found. Creating new log.
I20260619 04:35:29.342896 12214 log.cc:824] T 99c8ee2140b747a3874f8fbd116bfd60 P 010078eddda04c98a4894df9d73fd630: Log is configured to *not* fsync() on all Append() calls
I20260619 04:35:29.344884 12215 tablet_bootstrap.cc:654] T 99c8ee2140b747a3874f8fbd116bfd60 P 8ef28c8d03ca4f46acee4b5a97e6e9e9: Neither blocks nor log segments found. Creating new log.
I20260619 04:35:29.345417 12214 tablet_bootstrap.cc:492] T 99c8ee2140b747a3874f8fbd116bfd60 P 010078eddda04c98a4894df9d73fd630: No bootstrap required, opened a new log
I20260619 04:35:29.345870 12214 ts_tablet_manager.cc:1397] T 99c8ee2140b747a3874f8fbd116bfd60 P 010078eddda04c98a4894df9d73fd630: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20260619 04:35:29.345995 12215 log.cc:824] T 99c8ee2140b747a3874f8fbd116bfd60 P 8ef28c8d03ca4f46acee4b5a97e6e9e9: Log is configured to *not* fsync() on all Append() calls
I20260619 04:35:29.351948 12218 tablet_bootstrap.cc:492] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd: Bootstrap starting.
I20260619 04:35:29.354480 12215 tablet_bootstrap.cc:492] T 99c8ee2140b747a3874f8fbd116bfd60 P 8ef28c8d03ca4f46acee4b5a97e6e9e9: No bootstrap required, opened a new log
I20260619 04:35:29.354722 12215 ts_tablet_manager.cc:1397] T 99c8ee2140b747a3874f8fbd116bfd60 P 8ef28c8d03ca4f46acee4b5a97e6e9e9: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260619 04:35:29.354605 12214 raft_consensus.cc:348] T 99c8ee2140b747a3874f8fbd116bfd60 P 010078eddda04c98a4894df9d73fd630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010078eddda04c98a4894df9d73fd630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40213 } } peers { permanent_uuid: "8ef28c8d03ca4f46acee4b5a97e6e9e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33821 } } peers { permanent_uuid: "659be81aeb1d446988584cb79dd4f0fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34889 } }
I20260619 04:35:29.355063 12214 raft_consensus.cc:374] T 99c8ee2140b747a3874f8fbd116bfd60 P 010078eddda04c98a4894df9d73fd630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:35:29.355172 12214 raft_consensus.cc:729] T 99c8ee2140b747a3874f8fbd116bfd60 P 010078eddda04c98a4894df9d73fd630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 010078eddda04c98a4894df9d73fd630, State: Initialized, Role: FOLLOWER
I20260619 04:35:29.355502 12218 tablet_bootstrap.cc:654] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd: Neither blocks nor log segments found. Creating new log.
I20260619 04:35:29.355847 12214 consensus_queue.cc:260] T 99c8ee2140b747a3874f8fbd116bfd60 P 010078eddda04c98a4894df9d73fd630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010078eddda04c98a4894df9d73fd630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40213 } } peers { permanent_uuid: "8ef28c8d03ca4f46acee4b5a97e6e9e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33821 } } peers { permanent_uuid: "659be81aeb1d446988584cb79dd4f0fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34889 } }
I20260619 04:35:29.357414 12218 log.cc:824] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd: Log is configured to *not* fsync() on all Append() calls
I20260619 04:35:29.358242 12214 ts_tablet_manager.cc:1428] T 99c8ee2140b747a3874f8fbd116bfd60 P 010078eddda04c98a4894df9d73fd630: Time spent starting tablet: real 0.012s user 0.003s sys 0.007s
I20260619 04:35:29.361028 12218 tablet_bootstrap.cc:492] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd: No bootstrap required, opened a new log
I20260619 04:35:29.361294 12218 ts_tablet_manager.cc:1397] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260619 04:35:29.363646 12215 raft_consensus.cc:348] T 99c8ee2140b747a3874f8fbd116bfd60 P 8ef28c8d03ca4f46acee4b5a97e6e9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010078eddda04c98a4894df9d73fd630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40213 } } peers { permanent_uuid: "8ef28c8d03ca4f46acee4b5a97e6e9e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33821 } } peers { permanent_uuid: "659be81aeb1d446988584cb79dd4f0fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34889 } }
I20260619 04:35:29.363955 12215 raft_consensus.cc:374] T 99c8ee2140b747a3874f8fbd116bfd60 P 8ef28c8d03ca4f46acee4b5a97e6e9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:35:29.364042 12215 raft_consensus.cc:729] T 99c8ee2140b747a3874f8fbd116bfd60 P 8ef28c8d03ca4f46acee4b5a97e6e9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ef28c8d03ca4f46acee4b5a97e6e9e9, State: Initialized, Role: FOLLOWER
I20260619 04:35:29.364676 12215 consensus_queue.cc:260] T 99c8ee2140b747a3874f8fbd116bfd60 P 8ef28c8d03ca4f46acee4b5a97e6e9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010078eddda04c98a4894df9d73fd630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40213 } } peers { permanent_uuid: "8ef28c8d03ca4f46acee4b5a97e6e9e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33821 } } peers { permanent_uuid: "659be81aeb1d446988584cb79dd4f0fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34889 } }
I20260619 04:35:29.365672 12195 heartbeater.cc:503] Master 127.0.97.62:35503 was elected leader, sending a full tablet report...
I20260619 04:35:29.366205 12215 ts_tablet_manager.cc:1428] T 99c8ee2140b747a3874f8fbd116bfd60 P 8ef28c8d03ca4f46acee4b5a97e6e9e9: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260619 04:35:29.370196 12218 raft_consensus.cc:348] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010078eddda04c98a4894df9d73fd630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40213 } } peers { permanent_uuid: "8ef28c8d03ca4f46acee4b5a97e6e9e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33821 } } peers { permanent_uuid: "659be81aeb1d446988584cb79dd4f0fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34889 } }
I20260619 04:35:29.370537 12218 raft_consensus.cc:374] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:35:29.370604 12218 raft_consensus.cc:729] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 659be81aeb1d446988584cb79dd4f0fd, State: Initialized, Role: FOLLOWER
I20260619 04:35:29.371112 12218 consensus_queue.cc:260] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010078eddda04c98a4894df9d73fd630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40213 } } peers { permanent_uuid: "8ef28c8d03ca4f46acee4b5a97e6e9e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33821 } } peers { permanent_uuid: "659be81aeb1d446988584cb79dd4f0fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34889 } }
I20260619 04:35:29.372170 12061 heartbeater.cc:503] Master 127.0.97.62:35503 was elected leader, sending a full tablet report...
I20260619 04:35:29.372679 12218 ts_tablet_manager.cc:1428] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd: Time spent starting tablet: real 0.011s user 0.009s sys 0.003s
W20260619 04:35:29.454321 12062 tablet.cc:2369] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:35:29.481238 12196 tablet.cc:2369] T 99c8ee2140b747a3874f8fbd116bfd60 P 8ef28c8d03ca4f46acee4b5a97e6e9e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:35:29.489980 11919 tablet.cc:2369] T 99c8ee2140b747a3874f8fbd116bfd60 P 010078eddda04c98a4894df9d73fd630: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:35:29.632907 12222 raft_consensus.cc:482] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:35:29.633196 12222 raft_consensus.cc:504] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010078eddda04c98a4894df9d73fd630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40213 } } peers { permanent_uuid: "8ef28c8d03ca4f46acee4b5a97e6e9e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33821 } } peers { permanent_uuid: "659be81aeb1d446988584cb79dd4f0fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34889 } }
I20260619 04:35:29.634764 12222 leader_election.cc:288] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 010078eddda04c98a4894df9d73fd630 (127.0.97.1:40213), 8ef28c8d03ca4f46acee4b5a97e6e9e9 (127.0.97.3:33821)
I20260619 04:35:29.643656 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:35:29.635151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50753 (local address 127.0.97.3:33821)
0619 04:35:29.635337 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 04:35:29.635344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:35:29.636419 (+ 1075us) server_negotiation.cc:407] Connection header received
0619 04:35:29.636639 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:35:29.636645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:35:29.636730 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:35:29.636817 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:35:29.637480 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.638567 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.641348 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.642553 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.642678 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:35:29.642783 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 04:35:29.643278 (+ 495us) server_negotiation.cc:299] Negotiation successful
0619 04:35:29.643408 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:35:29.643663 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:35:29.635259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48139 (local address 127.0.97.1:40213)
0619 04:35:29.635531 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 04:35:29.635538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:35:29.636444 (+ 906us) server_negotiation.cc:407] Connection header received
0619 04:35:29.636639 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:35:29.636645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:35:29.636723 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:35:29.636817 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:35:29.637509 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.638552 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.641335 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.642554 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.642679 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:35:29.642783 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 04:35:29.643275 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 04:35:29.643408 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260619 04:35:29.643889 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:35:29.635117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40213 (local address 127.0.97.2:48139)
0619 04:35:29.636352 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0619 04:35:29.636366 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 04:35:29.636516 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:35:29.636834 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:35:29.636839 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:35:29.636869 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:35:29.637362 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:35:29.637371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.638673 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.638677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:35:29.641171 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:35:29.641185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.642798 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.642805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:35:29.643044 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:35:29.643156 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 04:35:29.643243 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 04:35:29.643432 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1142,"mutex_wait_us":252,"spinlock_wait_cycles":91264,"thread_start_us":865,"threads_started":1}
I20260619 04:35:29.644198 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:35:29.635037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33821 (local address 127.0.97.2:50753)
0619 04:35:29.636297 (+ 1260us) negotiation.cc:107] Waiting for socket to connect
0619 04:35:29.636317 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:35:29.636508 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:35:29.636834 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:35:29.636839 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:35:29.636876 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:35:29.637324 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:35:29.637340 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.638671 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.638676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:35:29.641170 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:35:29.641185 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.642803 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.642808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:35:29.643045 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:35:29.643154 (+ 109us) client_negotiation.cc:769] Sending connection context
0619 04:35:29.643239 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 04:35:29.643432 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1131,"mutex_wait_us":231,"spinlock_wait_cycles":3328,"thread_start_us":932,"threads_started":1}
I20260619 04:35:29.644897 11873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c8ee2140b747a3874f8fbd116bfd60" candidate_uuid: "659be81aeb1d446988584cb79dd4f0fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "010078eddda04c98a4894df9d73fd630" is_pre_election: true
I20260619 04:35:29.645154 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c8ee2140b747a3874f8fbd116bfd60" candidate_uuid: "659be81aeb1d446988584cb79dd4f0fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ef28c8d03ca4f46acee4b5a97e6e9e9" is_pre_election: true
I20260619 04:35:29.645368 11873 raft_consensus.cc:2454] T 99c8ee2140b747a3874f8fbd116bfd60 P 010078eddda04c98a4894df9d73fd630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 659be81aeb1d446988584cb79dd4f0fd in term 0.
I20260619 04:35:29.645627 12150 raft_consensus.cc:2454] T 99c8ee2140b747a3874f8fbd116bfd60 P 8ef28c8d03ca4f46acee4b5a97e6e9e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 659be81aeb1d446988584cb79dd4f0fd in term 0.
I20260619 04:35:29.646059 11952 leader_election.cc:302] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 010078eddda04c98a4894df9d73fd630, 659be81aeb1d446988584cb79dd4f0fd; no voters:
I20260619 04:35:29.646380 12222 raft_consensus.cc:2781] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:35:29.646507 12222 raft_consensus.cc:482] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:35:29.646600 12222 raft_consensus.cc:3037] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:35:29.649158 12222 raft_consensus.cc:504] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010078eddda04c98a4894df9d73fd630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40213 } } peers { permanent_uuid: "8ef28c8d03ca4f46acee4b5a97e6e9e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33821 } } peers { permanent_uuid: "659be81aeb1d446988584cb79dd4f0fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34889 } }
I20260619 04:35:29.649737 12222 leader_election.cc:288] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd [CANDIDATE]: Term 1 election: Requested vote from peers 010078eddda04c98a4894df9d73fd630 (127.0.97.1:40213), 8ef28c8d03ca4f46acee4b5a97e6e9e9 (127.0.97.3:33821)
I20260619 04:35:29.650158 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c8ee2140b747a3874f8fbd116bfd60" candidate_uuid: "659be81aeb1d446988584cb79dd4f0fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ef28c8d03ca4f46acee4b5a97e6e9e9"
I20260619 04:35:29.650182 11873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c8ee2140b747a3874f8fbd116bfd60" candidate_uuid: "659be81aeb1d446988584cb79dd4f0fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "010078eddda04c98a4894df9d73fd630"
I20260619 04:35:29.650324 12150 raft_consensus.cc:3037] T 99c8ee2140b747a3874f8fbd116bfd60 P 8ef28c8d03ca4f46acee4b5a97e6e9e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:35:29.650348 11873 raft_consensus.cc:3037] T 99c8ee2140b747a3874f8fbd116bfd60 P 010078eddda04c98a4894df9d73fd630 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:35:29.652331 12150 raft_consensus.cc:2454] T 99c8ee2140b747a3874f8fbd116bfd60 P 8ef28c8d03ca4f46acee4b5a97e6e9e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 659be81aeb1d446988584cb79dd4f0fd in term 1.
I20260619 04:35:29.652401 11873 raft_consensus.cc:2454] T 99c8ee2140b747a3874f8fbd116bfd60 P 010078eddda04c98a4894df9d73fd630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 659be81aeb1d446988584cb79dd4f0fd in term 1.
I20260619 04:35:29.652832 11952 leader_election.cc:302] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 010078eddda04c98a4894df9d73fd630, 659be81aeb1d446988584cb79dd4f0fd; no voters:
I20260619 04:35:29.653095 12222 raft_consensus.cc:2781] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:35:29.653484 12222 raft_consensus.cc:686] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd [term 1 LEADER]: Becoming Leader. State: Replica: 659be81aeb1d446988584cb79dd4f0fd, State: Running, Role: LEADER
I20260619 04:35:29.653918 12222 consensus_queue.cc:237] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010078eddda04c98a4894df9d73fd630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40213 } } peers { permanent_uuid: "8ef28c8d03ca4f46acee4b5a97e6e9e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33821 } } peers { permanent_uuid: "659be81aeb1d446988584cb79dd4f0fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34889 } }
I20260619 04:35:29.658197 11704 catalog_manager.cc:5495] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 659be81aeb1d446988584cb79dd4f0fd (127.0.97.2). New cstate: current_term: 1 leader_uuid: "659be81aeb1d446988584cb79dd4f0fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010078eddda04c98a4894df9d73fd630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ef28c8d03ca4f46acee4b5a97e6e9e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "659be81aeb1d446988584cb79dd4f0fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34889 } health_report { overall_health: HEALTHY } } }
I20260619 04:35:29.759666 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:35:29.754707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40062 (local address 127.0.97.2:34889)
0619 04:35:29.754879 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 04:35:29.754883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:35:29.754900 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:35:29.754982 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:35:29.754985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:35:29.755082 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:35:29.755197 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:35:29.755479 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.756189 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.757635 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.758147 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.758237 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:35:29.758681 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:35:29.759212 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:35:29.759312 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 04:35:29.759388 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 04:35:29.759451 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 04:35:29.767655 12150 raft_consensus.cc:1261] T 99c8ee2140b747a3874f8fbd116bfd60 P 8ef28c8d03ca4f46acee4b5a97e6e9e9 [term 1 FOLLOWER]: Refusing update from remote peer 659be81aeb1d446988584cb79dd4f0fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:35:29.767655 11873 raft_consensus.cc:1261] T 99c8ee2140b747a3874f8fbd116bfd60 P 010078eddda04c98a4894df9d73fd630 [term 1 FOLLOWER]: Refusing update from remote peer 659be81aeb1d446988584cb79dd4f0fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:35:29.768688 12225 consensus_queue.cc:1035] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "010078eddda04c98a4894df9d73fd630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:35:29.769021 12222 consensus_queue.cc:1035] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef28c8d03ca4f46acee4b5a97e6e9e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:35:29.779301 12228 mvcc.cc:205] Tried to move back new op lower bound from 7298431917118455808 to 7298431916665909248. Current Snapshot: MvccSnapshot[applied={T|T < 7298431917118455808}]
I20260619 04:35:29.791405 11705 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:35:29.791635 11705 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:35:29.795127 11705 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 889fc7d1dd844080847c809ec6e5509c: Sending DeleteTablet for 3 replicas of tablet 99c8ee2140b747a3874f8fbd116bfd60
I20260619 04:35:29.796196 12130 tablet_service.cc:1514] Processing DeleteTablet for tablet 99c8ee2140b747a3874f8fbd116bfd60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:35:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57850
I20260619 04:35:29.796255 11996 tablet_service.cc:1514] Processing DeleteTablet for tablet 99c8ee2140b747a3874f8fbd116bfd60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:35:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40046
I20260619 04:35:29.796577 11851 tablet_service.cc:1514] Processing DeleteTablet for tablet 99c8ee2140b747a3874f8fbd116bfd60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:35:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37414
I20260619 04:35:29.798435 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 11785
I20260619 04:35:29.817337 12238 tablet_replica.cc:330] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd: stopping tablet replica
I20260619 04:35:29.817880 12238 raft_consensus.cc:2229] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:35:29.818666 12238 raft_consensus.cc:2258] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:35:29.818884 12239 tablet_replica.cc:330] T 99c8ee2140b747a3874f8fbd116bfd60 P 010078eddda04c98a4894df9d73fd630: stopping tablet replica
I20260619 04:35:29.819396 12239 raft_consensus.cc:2229] T 99c8ee2140b747a3874f8fbd116bfd60 P 010078eddda04c98a4894df9d73fd630 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:35:29.820866 12239 raft_consensus.cc:2258] T 99c8ee2140b747a3874f8fbd116bfd60 P 010078eddda04c98a4894df9d73fd630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:35:29.821075 12238 ts_tablet_manager.cc:1905] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:35:29.825196 12237 tablet_replica.cc:330] T 99c8ee2140b747a3874f8fbd116bfd60 P 8ef28c8d03ca4f46acee4b5a97e6e9e9: stopping tablet replica
I20260619 04:35:29.825574 12237 raft_consensus.cc:2229] T 99c8ee2140b747a3874f8fbd116bfd60 P 8ef28c8d03ca4f46acee4b5a97e6e9e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:35:29.826036 12237 raft_consensus.cc:2258] T 99c8ee2140b747a3874f8fbd116bfd60 P 8ef28c8d03ca4f46acee4b5a97e6e9e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:35:29.827205 12239 ts_tablet_manager.cc:1905] T 99c8ee2140b747a3874f8fbd116bfd60 P 010078eddda04c98a4894df9d73fd630: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 04:35:29 dist-test-slave-n3tq krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 04:35:29.829064 12238 ts_tablet_manager.cc:1918] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:35:29.829226 12238 log.cc:1196] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/99c8ee2140b747a3874f8fbd116bfd60
I20260619 04:35:29.829739 12238 ts_tablet_manager.cc:1939] T 99c8ee2140b747a3874f8fbd116bfd60 P 659be81aeb1d446988584cb79dd4f0fd: Deleting consensus metadata
I20260619 04:35:29.831604 11692 catalog_manager.cc:4841] TS 659be81aeb1d446988584cb79dd4f0fd (127.0.97.2:34889): tablet 99c8ee2140b747a3874f8fbd116bfd60 (table test-table [id=4350c218d5534f058f6f00e26d97afba]) successfully deleted
I20260619 04:35:29.832340 12239 ts_tablet_manager.cc:1918] T 99c8ee2140b747a3874f8fbd116bfd60 P 010078eddda04c98a4894df9d73fd630: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:35:29.832525 12239 log.cc:1196] T 99c8ee2140b747a3874f8fbd116bfd60 P 010078eddda04c98a4894df9d73fd630: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-0/wal/wals/99c8ee2140b747a3874f8fbd116bfd60
I20260619 04:35:29.834107 12239 ts_tablet_manager.cc:1939] T 99c8ee2140b747a3874f8fbd116bfd60 P 010078eddda04c98a4894df9d73fd630: Deleting consensus metadata
I20260619 04:35:29.834199 12237 ts_tablet_manager.cc:1905] T 99c8ee2140b747a3874f8fbd116bfd60 P 8ef28c8d03ca4f46acee4b5a97e6e9e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:35:29.836179 11692 catalog_manager.cc:4841] TS 010078eddda04c98a4894df9d73fd630 (127.0.97.1:40213): tablet 99c8ee2140b747a3874f8fbd116bfd60 (table test-table [id=4350c218d5534f058f6f00e26d97afba]) successfully deleted
I20260619 04:35:29.839820 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:35:29.799990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37426 (local address 127.0.97.1:40213)
0619 04:35:29.800181 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 04:35:29.800187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:35:29.810425 (+ 10238us) server_negotiation.cc:407] Connection header received
0619 04:35:29.810935 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:35:29.810942 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:35:29.811077 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:35:29.811156 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:35:29.813112 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.814397 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.817064 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.817761 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.817862 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:35:29.832623 (+ 14761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:35:29.832637 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:35:29.832673 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:35:29.832724 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:35:29.836366 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:35:29.837314 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:35:29.837321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:35:29.837324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:35:29.837451 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:35:29.837856 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:35:29.837868 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:35:29.837870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:35:29.838190 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:35:29.838476 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 04:35:29.838830 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 04:35:29.839038 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:35:29.840519 11913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:35:29.840934 12237 ts_tablet_manager.cc:1918] T 99c8ee2140b747a3874f8fbd116bfd60 P 8ef28c8d03ca4f46acee4b5a97e6e9e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:35:29.841091 12237 log.cc:1196] T 99c8ee2140b747a3874f8fbd116bfd60 P 8ef28c8d03ca4f46acee4b5a97e6e9e9: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/99c8ee2140b747a3874f8fbd116bfd60
I20260619 04:35:29.841542 12237 ts_tablet_manager.cc:1939] T 99c8ee2140b747a3874f8fbd116bfd60 P 8ef28c8d03ca4f46acee4b5a97e6e9e9: Deleting consensus metadata
I20260619 04:35:29.843324 11689 catalog_manager.cc:4841] TS 8ef28c8d03ca4f46acee4b5a97e6e9e9 (127.0.97.3:33821): tablet 99c8ee2140b747a3874f8fbd116bfd60 (table test-table [id=4350c218d5534f058f6f00e26d97afba]) successfully deleted
I20260619 04:35:29.961169 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 11785
I20260619 04:35:29.975986 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 11924
Jun 19 04:35:29 dist-test-slave-n3tq krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 04:35:29.992367 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:35:29.976773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40066 (local address 127.0.97.2:34889)
0619 04:35:29.976956 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 04:35:29.976962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:35:29.977016 (+ 54us) server_negotiation.cc:407] Connection header received
0619 04:35:29.977177 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:35:29.977184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:35:29.977258 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:35:29.977365 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:35:29.978461 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.979696 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.981151 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:29.981803 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:29.981931 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:35:29.985952 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:35:29.985971 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:35:29.985991 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:35:29.986042 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:35:29.989734 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:35:29.990333 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:35:29.990342 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:35:29.990346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:35:29.990501 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:35:29.990807 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:35:29.990812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:35:29.990815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:35:29.991243 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:35:29.991560 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 04:35:29.991773 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 04:35:29.991978 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 04:35:29.993096 12056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:35:30.112589 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 11924
I20260619 04:35:30.126500 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 12067
Jun 19 04:35:30 dist-test-slave-n3tq krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 04:35:30.140364 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:35:30.127176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57856 (local address 127.0.97.3:33821)
0619 04:35:30.127338 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 04:35:30.127343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:35:30.127360 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:35:30.127521 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:35:30.127525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:35:30.127584 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:35:30.127661 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:35:30.128906 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:30.129602 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:30.130745 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:35:30.131115 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:35:30.131201 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:35:30.135402 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:35:30.135417 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:35:30.135433 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:35:30.135483 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:35:30.138078 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:35:30.138716 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:35:30.138722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:35:30.138724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:35:30.138859 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:35:30.139158 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:35:30.139161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:35:30.139163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:35:30.139501 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:35:30.139695 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 04:35:30.139915 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 04:35:30.140065 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 04:35:30.141011 12190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:35:30.230418 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 12067
I20260619 04:35:30.243017 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 11673
I20260619 04:35:30.244114 11735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:35:30.354069 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 11673
I20260619 04:35:30.370899 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 04:35:30.705474 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 04:35:30.705693 388 mini_ranger.cc:66] Stopping Ranger...
I20260619 04:35:31.040587 388 mini_ranger.cc:68] Stopped Ranger
2026-06-19 04:35:31.040 UTC [11259] LOG: received smart shutdown request
2026-06-19 04:35:31.043 UTC [11259] LOG: background worker "logical replication launcher" (PID 11268) exited with exit code 1
2026-06-19 04:35:31.044 UTC [11263] LOG: shutting down
2026-06-19 04:35:31.181 UTC [11259] LOG: database system is shut down
2026-06-19T04:35:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47291 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:35:31 dist-test-slave-n3tq krb5kdc[12253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:35:31 dist-test-slave-n3tq krb5kdc[12253](info): set up 2 sockets
Jun 19 04:35:31 dist-test-slave-n3tq krb5kdc[12253](info): commencing operation
krb5kdc: starting...
W20260619 04:35:32.304545 388 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:35:32 dist-test-slave-n3tq krb5kdc[12253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:35:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:35:32Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260619 04:35:32.592765 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task24iQRF/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/postgres -l logfile start
2026-06-19 04:35:34.798 UTC [12283] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 04:35:34.798 UTC [12283] LOG: listening on IPv4 address "127.0.97.20", port 51321
2026-06-19 04:35:34.801 UTC [12283] LOG: listening on Unix socket "/tmp/.s.PGSQL.51321"
2026-06-19 04:35:34.819 UTC [12286] LOG: database system was shut down at 2026-06-19 04:35:33 UTC
2026-06-19 04:35:34.822 UTC [12283] LOG: database system is ready to accept connections
I20260619 04:35:35.708212 388 mini_postgres.cc:96] Postgres bound to 51321
2026-06-19 04:35:35.723 UTC [12294] FATAL: database "slave" does not exist
127.0.97.20:51321 - accepting connections
I20260619 04:35:35.724407 388 mini_ranger.cc:161] Starting Ranger...
I20260619 04:35:35.748636 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 04:35:36.130859 388 mini_ranger.cc:87] Created ranger Postgres database
I20260619 04:35:36.130946 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ranger-admin
2026-06-19 04:35:36,398 [I] DB FLAVOR :POSTGRES
2026-06-19 04:35:36,398 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 04:35:36,398 [I] Checking connection..
2026-06-19 04:35:36,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 04:35:36,653 [I] Checking connection passed.
2026-06-19 04:35:36,653 [I] --------- Verifying version history table ---------
2026-06-19 04:35:36,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 04:35:36,897 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 04:35:36,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 04:35:37,132 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 04:35:37,132 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 04:35:37,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 04:35:37.379 UTC [12358] WARNING: there is no transaction in progress
2026-06-19 04:35:37,384 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 04:35:37,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 04:35:37,634 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 04:35:37,634 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 04:35:37,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 04:35:37,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 04:35:38,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-n3tq.c.gcp-upstream.internal','N') ;"
2026-06-19 04:35:38,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 04:35:38,609 [I] Table x_portal_user does not exist in database ranger
2026-06-19 04:35:38,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 04:35:38,850 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 04:35:38,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 04:35:39,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 04:35:39,332 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 04:35:39,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 04:35:40.204 UTC [12466] WARNING: there is no transaction in progress
2026-06-19 04:35:40.220 UTC [12466] WARNING: there is no transaction in progress
2026-06-19 04:35:40.234 UTC [12466] WARNING: there is no transaction in progress
2026-06-19 04:35:40.248 UTC [12466] WARNING: there is no transaction in progress
2026-06-19 04:35:40.262 UTC [12466] WARNING: there is no transaction in progress
2026-06-19 04:35:40.280 UTC [12466] WARNING: there is no transaction in progress
2026-06-19 04:35:40.343 UTC [12466] WARNING: there is no transaction in progress
2026-06-19 04:35:40.349 UTC [12466] WARNING: there is no transaction in progress
2026-06-19 04:35:40.363 UTC [12466] WARNING: there is no transaction in progress
2026-06-19 04:35:40.372 UTC [12466] WARNING: there is no transaction in progress
2026-06-19 04:35:40.381 UTC [12466] WARNING: there is no transaction in progress
2026-06-19 04:35:40.394 UTC [12466] WARNING: there is no transaction in progress
2026-06-19 04:35:40.403 UTC [12466] WARNING: there is no transaction in progress
2026-06-19 04:35:40.409 UTC [12466] WARNING: there is no transaction in progress
2026-06-19 04:35:40.415 UTC [12466] WARNING: there is no transaction in progress
2026-06-19 04:35:40.421 UTC [12466] WARNING: there is no transaction in progress
2026-06-19 04:35:40,979 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 04:35:40,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 04:35:41,227 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 04:35:41,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 04:35:41,477 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 04:35:41,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 04:35:41,699 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 04:35:41,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 04:35:41,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 04:35:42,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-n3tq.c.gcp-upstream.internal';"
2026-06-19 04:35:42,394 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 04:35:42,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 04:35:42,648 [I] DB_PATCHES have already been applied
I20260619 04:35:42.654503 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ranger-admin:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task24iQRF/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 04:35:42.654594 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260619 04:35:42.667829 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:44671
Jun 19, 2026 4:35:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:35:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:35:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 4:35:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 4:35:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 4:35:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 4:35:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 04:35:43.695282 388 mini_ranger.cc:160] Time spent starting Ranger: real 7.971s user 0.003s sys 0.035s
Jun 19, 2026 4:36:02 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 4:36:02 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 4:36:02 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 4:36:02 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 4:36:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 4:36:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 4:36:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 4:36:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 4:36:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 4:36:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 4:36:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 4:36:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 4:36:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 4:36:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 4:36:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 4:36:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 4:36:02 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 4:36:03 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 04:36:04.194262 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 04:36:04 dist-test-slave-n3tq krb5kdc[12253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843764, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 04:36:04.359054 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 04:36:04.378173 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 04:36:05.111624 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 04:36:05.111716 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ranger-kms
I20260619 04:36:05.111750 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:51321
2026-06-19 04:36:05,431 [I] DB FLAVOR :POSTGRES
2026-06-19 04:36:05,432 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 04:36:05,432 [I] Checking connection
2026-06-19 04:36:05,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 04:36:05,655 [I] connection success
2026-06-19 04:36:05,656 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 04:36:05,656 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 04:36:05,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 04:36:05,879 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 04:36:05,879 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 04:36:05,879 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 04:36:05,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51321/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 04:36:06,134 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 04:36:06.575740 388 mini_ranger_kms.cc:310] Created kms service
I20260619 04:36:06.768803 388 mini_ranger_kms.cc:326] Created kudu user
I20260619 04:36:06.817503 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 04:36:06 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843764, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 04:36:07.207741 388 mini_ranger_kms.cc:384] Added ranger policy
I20260619 04:36:07.208048 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ranger-kms:/tmp/dist-test-task24iQRF/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task24iQRF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 04:36:07.208091 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260619 04:36:07.224995 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12646
I20260619 04:36:07.225104 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:36311
Jun 19, 2026 4:36:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:36:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:36:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 4:36:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 4:36:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 4:36:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 04:36:09.310313 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.951s user 0.085s sys 0.246s
I20260619 04:36:09.310508 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 04:36:09.310657 388 mini_ranger_kms.cc:401] 127.0.97.20:36311/kms/v1/keys
Jun 19 04:36:09 dist-test-slave-n3tq krb5kdc[12253](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781843769, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 04:36:09 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843769, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 04:36:10 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843769, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 04:36:12 dist-test-slave-n3tq krb5kdc[12253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:36:12.335583 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41715
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41715
--ranger_config_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:36311/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:36:12.646242 12697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:12.646528 12697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:12.646656 12697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:12.653779 12697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:36:12.653901 12697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:12.653972 12697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:12.654047 12697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:36:12.654114 12697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:36:12.664630 12697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:36311/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41715
--ranger_config_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:12.665889 12697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:12.667189 12697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:12.678187 12702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:12.678249 12705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:12.678272 12703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:12.678982 12697 server_base.cc:1034] running on GCE node
I20260619 04:36:12.679710 12697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:12.681209 12697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:12.682410 12697 hybrid_clock.cc:648] HybridClock initialized: now 1781843772682373 us; error 28 us; skew 500 ppm
I20260619 04:36:12.682736 12697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:12 dist-test-slave-n3tq krb5kdc[12253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:12.697957 12697 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:36:12.699527 12697 webserver.cc:460] Webserver started at http://127.0.97.62:44407/ using document root <none> and password file <none>
I20260619 04:36:12.700228 12697 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:12.700342 12697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:12.700678 12697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 04:36:12 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 04:36:12.756743 12697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "0046ff439fa446828fcdc747e62956b3"
format_stamp: "Formatted at 2026-06-19 04:36:12 on dist-test-slave-n3tq"
server_key: "c983f3368c05aa747f3bbbbf329ba25b"
server_key_iv: "b114afee7530ec525f200fe35c3849ad"
server_key_version: "kuduclusterkey@0"
I20260619 04:36:12.757553 12697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "0046ff439fa446828fcdc747e62956b3"
format_stamp: "Formatted at 2026-06-19 04:36:12 on dist-test-slave-n3tq"
server_key: "c983f3368c05aa747f3bbbbf329ba25b"
server_key_iv: "b114afee7530ec525f200fe35c3849ad"
server_key_version: "kuduclusterkey@0"
I20260619 04:36:12.763248 12697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:36:12.779120 12712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:12.780686 12697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 04:36:12.780869 12697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "0046ff439fa446828fcdc747e62956b3"
format_stamp: "Formatted at 2026-06-19 04:36:12 on dist-test-slave-n3tq"
server_key: "c983f3368c05aa747f3bbbbf329ba25b"
server_key_iv: "b114afee7530ec525f200fe35c3849ad"
server_key_version: "kuduclusterkey@0"
I20260619 04:36:12.781076 12697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:12.808596 12697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:12.812355 12697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:12.812665 12697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:12.829241 12697 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41715
I20260619 04:36:12.829243 12763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41715 every 8 connection(s)
I20260619 04:36:12.830859 12697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:36:12.834540 12764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:36:12.838459 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 12697
I20260619 04:36:12.838649 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:36:12.839248 388 external_mini_cluster.cc:1428] Setting key e3a9d91ca62f805e5511919518b18871
I20260619 04:36:12.845764 12764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3: Bootstrap starting.
I20260619 04:36:12.849185 12764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3: Neither blocks nor log segments found. Creating new log.
I20260619 04:36:12.850342 12764 log.cc:824] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3: Log is configured to *not* fsync() on all Append() calls
I20260619 04:36:12.853107 12764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3: No bootstrap required, opened a new log
Jun 19 04:36:12 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:36:12.862138 12767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:12.841120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.0.97.62:41715)
0619 04:36:12.841871 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:12.841887 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:12.841938 (+ 51us) server_negotiation.cc:407] Connection header received
0619 04:36:12.844265 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:12.844293 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:12.844796 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:12.845196 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:12.846333 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:12.848208 (+ 1875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:12.849452 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:12.850434 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:12.850615 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:12.854385 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:12.854394 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:12.854411 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:12.854449 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:12.858060 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:12.858643 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:12.858651 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:12.858660 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:12.858843 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:12.859154 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:12.859159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:12.859162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:12.859822 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:12.860141 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:12.860408 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 04:36:12.860811 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":311,"threads_started":1}
I20260619 04:36:12.863168 12764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0046ff439fa446828fcdc747e62956b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41715 } }
I20260619 04:36:12.863576 12764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:36:12.863675 12764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0046ff439fa446828fcdc747e62956b3, State: Initialized, Role: FOLLOWER
I20260619 04:36:12.864198 12764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0046ff439fa446828fcdc747e62956b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41715 } }
I20260619 04:36:12.864398 12764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:36:12.864533 12764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:36:12.864698 12764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:36:12.866755 12764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0046ff439fa446828fcdc747e62956b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41715 } }
I20260619 04:36:12.867169 12764 leader_election.cc:302] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0046ff439fa446828fcdc747e62956b3; no voters:
I20260619 04:36:12.867633 12764 leader_election.cc:288] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:36:12.867753 12769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:36:12.868908 12769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [term 1 LEADER]: Becoming Leader. State: Replica: 0046ff439fa446828fcdc747e62956b3, State: Running, Role: LEADER
I20260619 04:36:12.869459 12769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0046ff439fa446828fcdc747e62956b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41715 } }
I20260619 04:36:12.869695 12764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:36:12.872179 12771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0046ff439fa446828fcdc747e62956b3. Latest consensus state: current_term: 1 leader_uuid: "0046ff439fa446828fcdc747e62956b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0046ff439fa446828fcdc747e62956b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41715 } } }
I20260619 04:36:12.872610 12771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [sys.catalog]: This master's current role is: LEADER
I20260619 04:36:12.872205 12770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0046ff439fa446828fcdc747e62956b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0046ff439fa446828fcdc747e62956b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41715 } } }
I20260619 04:36:12.873134 12770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [sys.catalog]: This master's current role is: LEADER
I20260619 04:36:12.874311 12778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:36:12.879107 12778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:36:12.879774 12764 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 04:36:12.897367 12778 catalog_manager.cc:1347] Generated new cluster ID: a9b25b1691b849afab96bc4c96ae28fd
I20260619 04:36:12.897574 12778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:36:12.913368 12778 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:36:12.915299 12778 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:36:12.929857 12778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3: Generated new TSK 0
I20260619 04:36:12.930946 12778 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 04:36:13.644289 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:36311/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41715
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:36:13 dist-test-slave-n3tq krb5kdc[12253](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781843773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 04:36:14.020244 12809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:14.020681 12809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:14.020850 12809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:14.032063 12809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:14.032208 12809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:14.032444 12809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:36:14.047683 12809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:36311/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:14.049664 12809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:14.051515 12809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:14.067854 12819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:14.069469 12820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:14.077839 12809 server_base.cc:1034] running on GCE node
W20260619 04:36:14.078105 12822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:14.078910 12809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:14.081002 12809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:14.082286 12809 hybrid_clock.cc:648] HybridClock initialized: now 1781843774082217 us; error 60 us; skew 500 ppm
I20260619 04:36:14.082794 12809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:14 dist-test-slave-n3tq krb5kdc[12253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:14.087034 12809 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:36:14.089941 12809 webserver.cc:460] Webserver started at http://127.0.97.1:36653/ using document root <none> and password file <none>
I20260619 04:36:14.090873 12809 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:14.091010 12809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:14.091364 12809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 04:36:14 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 04:36:14.115772 12809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "717b5617f19f463899b66ed4e343a864"
format_stamp: "Formatted at 2026-06-19 04:36:14 on dist-test-slave-n3tq"
server_key: "133419d2e4c788d54143238805986a70"
server_key_iv: "95a4482a4b53b1fa09abe5ba681fc0b2"
server_key_version: "kuduclusterkey@0"
I20260619 04:36:14.117033 12809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "717b5617f19f463899b66ed4e343a864"
format_stamp: "Formatted at 2026-06-19 04:36:14 on dist-test-slave-n3tq"
server_key: "133419d2e4c788d54143238805986a70"
server_key_iv: "95a4482a4b53b1fa09abe5ba681fc0b2"
server_key_version: "kuduclusterkey@0"
I20260619 04:36:14.130870 12809 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.007s sys 0.005s
I20260619 04:36:14.162788 12829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:14.165072 12809 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.003s sys 0.011s
I20260619 04:36:14.165380 12809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "717b5617f19f463899b66ed4e343a864"
format_stamp: "Formatted at 2026-06-19 04:36:14 on dist-test-slave-n3tq"
server_key: "133419d2e4c788d54143238805986a70"
server_key_iv: "95a4482a4b53b1fa09abe5ba681fc0b2"
server_key_version: "kuduclusterkey@0"
I20260619 04:36:14.165714 12809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:14.206828 12809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:14.211849 12809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:14.212212 12809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:14.225123 12809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:36:14.227057 12809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:36:14.227150 12809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:14.227288 12809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:36:14.227341 12809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:14.286489 12809 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37925
I20260619 04:36:14.286489 12941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37925 every 8 connection(s)
I20260619 04:36:14.287927 12809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:36:14.296370 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 12809
I20260619 04:36:14.296607 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:36:14.297403 388 external_mini_cluster.cc:1428] Setting key 391e33f8ceeda2ff6b6909a22fb2405a
Jun 19 04:36:14 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 04:36:14.345988 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:14.292335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47517 (local address 127.0.97.62:41715)
0619 04:36:14.315063 (+ 22728us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:14.315070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:14.317762 (+ 2692us) server_negotiation.cc:407] Connection header received
0619 04:36:14.326382 (+ 8620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:14.326388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:14.326504 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:14.326619 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:14.328840 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:14.329882 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:14.332195 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:14.332814 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:14.332909 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:14.338988 (+ 6079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:14.338996 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:14.339000 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:14.339054 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:14.342059 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:14.342956 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:14.342961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:14.342964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:14.343112 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:14.343920 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:14.343926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:14.343930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:14.344262 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:14.344385 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:14.345380 (+ 995us) server_negotiation.cc:299] Negotiation successful
0619 04:36:14.345624 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22564,"thread_start_us":22735,"threads_started":1}
I20260619 04:36:14.349639 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:14.291650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41715 (local address 127.0.97.1:47517)
0619 04:36:14.315862 (+ 24212us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:14.315934 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:14.319361 (+ 3427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:14.327084 (+ 7723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:14.327098 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:14.327722 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:14.328630 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:14.328660 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:14.330104 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:14.330110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:14.332022 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:14.332045 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:14.333254 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:14.333261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:14.333561 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:14.336024 (+ 2463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:14.336070 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:14.338681 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:14.342218 (+ 3537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:14.342227 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:14.342247 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:14.342812 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:14.343256 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:14.343260 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:14.343263 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:14.343780 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:14.344504 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:14.344513 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:14.344950 (+ 437us) client_negotiation.cc:769] Sending connection context
0619 04:36:14.346534 (+ 1584us) client_negotiation.cc:240] Negotiation successful
0619 04:36:14.347119 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23808,"thread_start_us":32933,"threads_started":1}
I20260619 04:36:14.352007 12942 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41715
I20260619 04:36:14.352312 12942 heartbeater.cc:464] Registering TS with master...
I20260619 04:36:14.353251 12942 heartbeater.cc:511] Master 127.0.97.62:41715 requested a full tablet report, sending...
I20260619 04:36:14.355691 12729 ts_manager.cc:194] Registered new tserver with Master: 717b5617f19f463899b66ed4e343a864 (127.0.97.1:37925)
I20260619 04:36:14.357589 12729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 04:36:14.375169 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:36311/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41715
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:36:14 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 04:36:14 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260619 04:36:14.887576 12948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:14.888092 12948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:14.888298 12948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:14.900283 12948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:14.900539 12948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:14.900791 12948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:36:14.916872 12948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:36311/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:14.918928 12948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:14.920646 12948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:14.952687 12955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:14.958192 12956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:14.966106 12948 server_base.cc:1034] running on GCE node
W20260619 04:36:14.966593 12958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:14.967257 12948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:14.968300 12948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:14.969544 12948 hybrid_clock.cc:648] HybridClock initialized: now 1781843774969487 us; error 72 us; skew 500 ppm
I20260619 04:36:14.969875 12948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:14 dist-test-slave-n3tq krb5kdc[12253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:14.973407 12948 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:36:14.993608 12948 webserver.cc:460] Webserver started at http://127.0.97.2:42689/ using document root <none> and password file <none>
I20260619 04:36:14.994387 12948 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:14.994520 12948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:14.994912 12948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 04:36:14 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 04:36:15.011409 12948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "f59e514c807d4105aca5434cbd5bfb64"
format_stamp: "Formatted at 2026-06-19 04:36:15 on dist-test-slave-n3tq"
server_key: "e032bcf62ca83602a983785dd6a7c69e"
server_key_iv: "928058b732b123c1dca064483c7cfbff"
server_key_version: "kuduclusterkey@0"
I20260619 04:36:15.012305 12948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f59e514c807d4105aca5434cbd5bfb64"
format_stamp: "Formatted at 2026-06-19 04:36:15 on dist-test-slave-n3tq"
server_key: "e032bcf62ca83602a983785dd6a7c69e"
server_key_iv: "928058b732b123c1dca064483c7cfbff"
server_key_version: "kuduclusterkey@0"
I20260619 04:36:15.017685 12948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 04:36:15.030997 12972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:15.032503 12948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:36:15.032677 12948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "f59e514c807d4105aca5434cbd5bfb64"
format_stamp: "Formatted at 2026-06-19 04:36:15 on dist-test-slave-n3tq"
server_key: "e032bcf62ca83602a983785dd6a7c69e"
server_key_iv: "928058b732b123c1dca064483c7cfbff"
server_key_version: "kuduclusterkey@0"
I20260619 04:36:15.032866 12948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:15.066912 12948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:15.070922 12948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:15.071262 12948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:15.072256 12948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:36:15.073803 12948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:36:15.073873 12948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:15.073983 12948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:36:15.074029 12948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:15.106702 12948 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40391
I20260619 04:36:15.106719 13084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40391 every 8 connection(s)
I20260619 04:36:15.107973 12948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:36:15.114073 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 12948
I20260619 04:36:15.114264 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:36:15.114727 388 external_mini_cluster.cc:1428] Setting key ca1896dc06821c2883a95277fc8decb4
Jun 19 04:36:15 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:36:15.130115 13087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:15.110589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48143 (local address 127.0.97.62:41715)
0619 04:36:15.111091 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:15.111097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:15.112140 (+ 1043us) server_negotiation.cc:407] Connection header received
0619 04:36:15.114705 (+ 2565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:15.114711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:15.114812 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:15.114939 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:15.116849 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:15.117923 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:15.120221 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:15.120850 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:15.120962 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:15.124617 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:15.124625 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:15.124641 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:15.124681 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:15.127168 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:15.127970 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:15.127974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:15.127976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:15.128087 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:15.128536 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:15.128539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:15.128542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:15.128815 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:15.128925 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:15.129685 (+ 760us) server_negotiation.cc:299] Negotiation successful
0619 04:36:15.129853 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":256,"threads_started":1}
I20260619 04:36:15.131443 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:15.111103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41715 (local address 127.0.97.2:48143)
0619 04:36:15.111965 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:15.112013 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:15.114377 (+ 2364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:15.115205 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:15.115217 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:15.115812 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:15.116654 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:15.116680 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:15.118092 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:15.118097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:15.120067 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:15.120090 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:15.121077 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:15.121085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:15.121384 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:15.122110 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:15.122132 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:15.124363 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:15.127305 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:15.127313 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:15.127329 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:15.127860 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:15.128182 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:15.128187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:15.128191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:15.128414 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:15.128931 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:15.128939 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:15.129331 (+ 392us) client_negotiation.cc:769] Sending connection context
0619 04:36:15.129609 (+ 278us) client_negotiation.cc:240] Negotiation successful
0619 04:36:15.130099 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":338,"threads_started":1}
I20260619 04:36:15.132863 13085 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41715
I20260619 04:36:15.133153 13085 heartbeater.cc:464] Registering TS with master...
I20260619 04:36:15.133826 13085 heartbeater.cc:511] Master 127.0.97.62:41715 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 04:36:15.135170 12729 ts_manager.cc:194] Registered new tserver with Master: f59e514c807d4105aca5434cbd5bfb64 (127.0.97.2:40391)
I20260619 04:36:15.136019 12729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 04:36:15.161283 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:36311/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41715
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:36:15.361712 12942 heartbeater.cc:503] Master 127.0.97.62:41715 was elected leader, sending a full tablet report...
W20260619 04:36:15.475386 13091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:15.475656 13091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:15.475785 13091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:15.482823 13091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:15.482936 13091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:15.483080 13091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:36:15.492532 13091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:36311/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:15.493752 13091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:15.494967 13091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:15.507033 13096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:15.506963 13097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:15.507014 13099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:15.507834 13091 server_base.cc:1034] running on GCE node
I20260619 04:36:15.508391 13091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:15.509303 13091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:15.510522 13091 hybrid_clock.cc:648] HybridClock initialized: now 1781843775510467 us; error 42 us; skew 500 ppm
I20260619 04:36:15.510844 13091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:15 dist-test-slave-n3tq krb5kdc[12253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:15.513746 13091 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:36:15.515046 13091 webserver.cc:460] Webserver started at http://127.0.97.3:34521/ using document root <none> and password file <none>
I20260619 04:36:15.515661 13091 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:15.515779 13091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:15.516043 13091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 04:36:15 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 04:36:15.532823 13091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "40d53f48115948f9ab52975a81406d60"
format_stamp: "Formatted at 2026-06-19 04:36:15 on dist-test-slave-n3tq"
server_key: "74f8b336d978210f94b6efc868daa9e0"
server_key_iv: "c41b7028a2abf2ee7845eefa617d9b4e"
server_key_version: "kuduclusterkey@0"
I20260619 04:36:15.533902 13091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "40d53f48115948f9ab52975a81406d60"
format_stamp: "Formatted at 2026-06-19 04:36:15 on dist-test-slave-n3tq"
server_key: "74f8b336d978210f94b6efc868daa9e0"
server_key_iv: "c41b7028a2abf2ee7845eefa617d9b4e"
server_key_version: "kuduclusterkey@0"
I20260619 04:36:15.540473 13091 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 04:36:15.556810 13106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:15.558346 13091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:36:15.558503 13091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "40d53f48115948f9ab52975a81406d60"
format_stamp: "Formatted at 2026-06-19 04:36:15 on dist-test-slave-n3tq"
server_key: "74f8b336d978210f94b6efc868daa9e0"
server_key_iv: "c41b7028a2abf2ee7845eefa617d9b4e"
server_key_version: "kuduclusterkey@0"
I20260619 04:36:15.558712 13091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:15.594105 13091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:15.597878 13091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:15.598150 13091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:15.599049 13091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:36:15.600378 13091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:36:15.600477 13091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:15.600603 13091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:36:15.600656 13091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:15.628705 13091 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38969
I20260619 04:36:15.628743 13218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38969 every 8 connection(s)
I20260619 04:36:15.629904 13091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:36:15.631013 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 13091
I20260619 04:36:15.631178 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:36:15.631745 388 external_mini_cluster.cc:1428] Setting key 5ed2991cf3520b25be9cc5e242f083ca
Jun 19 04:36:15 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:36:15.649137 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:15.632797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52123 (local address 127.0.97.62:41715)
0619 04:36:15.633250 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:15.633256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:15.634306 (+ 1050us) server_negotiation.cc:407] Connection header received
0619 04:36:15.636071 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:15.636075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:15.636169 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:15.636270 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:15.637812 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:15.638509 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:15.639989 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:15.640409 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:15.640531 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:15.643570 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:15.643574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:15.643577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:15.643607 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:15.646198 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:15.646925 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:15.646930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:15.646933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:15.647085 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:15.647467 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:15.647472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:15.647476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:15.647731 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:15.647877 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:15.648566 (+ 689us) server_negotiation.cc:299] Negotiation successful
0619 04:36:15.648789 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":317,"threads_started":1}
I20260619 04:36:15.650038 13222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:15.633407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41715 (local address 127.0.97.3:52123)
0619 04:36:15.634161 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:15.634202 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:15.635836 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:15.636537 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:15.636546 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:15.636945 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:15.637648 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:15.637674 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:15.638656 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:15.638662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:15.639859 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:15.639871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:15.640635 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:15.640642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:15.640869 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:15.641415 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:15.641434 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:15.643381 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:15.646320 (+ 2939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:15.646326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:15.646339 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:15.646821 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:15.647188 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:15.647193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:15.647198 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:15.647365 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:15.647851 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:15.647859 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:15.648212 (+ 353us) client_negotiation.cc:769] Sending connection context
0619 04:36:15.648471 (+ 259us) client_negotiation.cc:240] Negotiation successful
0619 04:36:15.648972 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":336,"threads_started":1}
I20260619 04:36:15.651288 13219 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41715
I20260619 04:36:15.651577 13219 heartbeater.cc:464] Registering TS with master...
I20260619 04:36:15.652263 13219 heartbeater.cc:511] Master 127.0.97.62:41715 requested a full tablet report, sending...
I20260619 04:36:15.653725 12729 ts_manager.cc:194] Registered new tserver with Master: 40d53f48115948f9ab52975a81406d60 (127.0.97.3:38969)
I20260619 04:36:15.654515 12729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52123
I20260619 04:36:15.659170 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:36:15.660730 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 12809
Jun 19 04:36:15 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 04:36:15.676359 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:15.661585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33778 (local address 127.0.97.1:37925)
0619 04:36:15.662200 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:15.662207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:15.662230 (+ 23us) server_negotiation.cc:407] Connection header received
0619 04:36:15.662291 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:15.662298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:15.662494 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:15.662614 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:15.664064 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:15.665152 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:15.666518 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:15.667023 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:15.667138 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:15.670934 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:15.670946 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:15.670959 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:15.670994 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:15.673827 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:15.674440 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:15.674446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:15.674449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:15.674612 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:15.674897 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:15.674900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:15.674902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:15.675202 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:15.675423 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:15.675820 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 04:36:15.676012 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":310,"threads_started":1}
I20260619 04:36:15.677591 12937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:15.759200 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 12809
I20260619 04:36:15.771833 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 12948
Jun 19 04:36:15 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 04:36:15.786913 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:15.772720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40282 (local address 127.0.97.2:40391)
0619 04:36:15.773183 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:15.773190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:15.773214 (+ 24us) server_negotiation.cc:407] Connection header received
0619 04:36:15.773283 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:15.773290 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:15.773498 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:15.773616 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:15.774843 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:15.775815 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:15.777163 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:15.777696 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:15.777791 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:15.781714 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:15.781726 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:15.781740 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:15.781776 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:15.784346 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:15.784997 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:15.785004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:15.785007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:15.785129 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:15.785577 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:15.785580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:15.785583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:15.785927 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:15.786156 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:15.786433 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 04:36:15.786633 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":210,"threads_started":1}
I20260619 04:36:15.788110 13080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:15.874790 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 12948
I20260619 04:36:15.886997 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 13091
Jun 19 04:36:15 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 04:36:15.902411 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:15.887869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55722 (local address 127.0.97.3:38969)
0619 04:36:15.888442 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:15.888470 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:15.888505 (+ 35us) server_negotiation.cc:407] Connection header received
0619 04:36:15.888563 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:15.888569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:15.888777 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:15.888916 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:15.890151 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:15.891151 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:15.892439 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:15.893074 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:15.893186 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:15.897210 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:15.897221 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:15.897235 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:15.897265 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:15.899910 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:15.900606 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:15.900613 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:15.900615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:15.900742 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:15.901077 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:15.901080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:15.901082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:15.901404 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:15.901633 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:15.901851 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 04:36:15.902094 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":284,"threads_started":1}
I20260619 04:36:15.903646 13214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:15.985067 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 13091
I20260619 04:36:15.997283 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 12697
I20260619 04:36:15.998476 12759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:16.121733 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 12697
I20260619 04:36:16.138538 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41715
--webserver_interface=127.0.97.62
--webserver_port=44407
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41715
--ranger_config_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:36311/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:36311/kms with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:36:16.445466 13231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:16.445722 13231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:16.445813 13231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:16.452787 13231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:36:16.452888 13231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:16.452937 13231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:16.452988 13231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:36:16.453032 13231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:36:16.462672 13231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:36311/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41715
--ranger_config_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:16.463840 13231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:16.465096 13231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:16.476480 13239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:16.476511 13236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:16.476545 13237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:16.477109 13231 server_base.cc:1034] running on GCE node
I20260619 04:36:16.477881 13231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:16.479283 13231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:16.480502 13231 hybrid_clock.cc:648] HybridClock initialized: now 1781843776480427 us; error 48 us; skew 500 ppm
I20260619 04:36:16.480844 13231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:16 dist-test-slave-n3tq krb5kdc[12253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:16.483726 13231 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:36:16.485087 13231 webserver.cc:460] Webserver started at http://127.0.97.62:44407/ using document root <none> and password file <none>
I20260619 04:36:16.485688 13231 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:16.485801 13231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 04:36:16 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 04:36:16.534065 13231 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 04:36:16.537011 13247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:16.538470 13231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:36:16.538645 13231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "0046ff439fa446828fcdc747e62956b3"
format_stamp: "Formatted at 2026-06-19 04:36:12 on dist-test-slave-n3tq"
server_key: "c983f3368c05aa747f3bbbbf329ba25b"
server_key_iv: "b114afee7530ec525f200fe35c3849ad"
server_key_version: "kuduclusterkey@0"
I20260619 04:36:16.539592 13231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:16.553272 13231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:16.557036 13231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:16.557272 13231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:16.574082 13231 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41715
I20260619 04:36:16.574103 13298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41715 every 8 connection(s)
I20260619 04:36:16.575506 13231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:36:16.577706 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 13231
I20260619 04:36:16.578665 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:36311/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:37925
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41715
--webserver_port=36653
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:36311/kms with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:36:16.584323 13299 sys_catalog.cc:263] Verifying existing consensus state
I20260619 04:36:16.588161 13299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3: Bootstrap starting.
I20260619 04:36:16.607367 13299 log.cc:824] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3: Log is configured to *not* fsync() on all Append() calls
I20260619 04:36:16.617447 13299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 04:36:16.617928 13299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3: Bootstrap complete.
I20260619 04:36:16.626475 13299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0046ff439fa446828fcdc747e62956b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41715 } }
I20260619 04:36:16.627305 13299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0046ff439fa446828fcdc747e62956b3, State: Initialized, Role: FOLLOWER
I20260619 04:36:16.627861 13299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0046ff439fa446828fcdc747e62956b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41715 } }
I20260619 04:36:16.628074 13299 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:36:16.628185 13299 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:36:16.628332 13299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [term 1 FOLLOWER]: Advancing to term 2
I20260619 04:36:16.631956 13299 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0046ff439fa446828fcdc747e62956b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41715 } }
I20260619 04:36:16.632422 13299 leader_election.cc:302] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0046ff439fa446828fcdc747e62956b3; no voters:
I20260619 04:36:16.633014 13299 leader_election.cc:288] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 04:36:16.633165 13303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 04:36:16.634081 13303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [term 2 LEADER]: Becoming Leader. State: Replica: 0046ff439fa446828fcdc747e62956b3, State: Running, Role: LEADER
I20260619 04:36:16.634719 13303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0046ff439fa446828fcdc747e62956b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41715 } }
I20260619 04:36:16.635063 13299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:36:16.637197 13304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0046ff439fa446828fcdc747e62956b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0046ff439fa446828fcdc747e62956b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41715 } } }
I20260619 04:36:16.637557 13304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [sys.catalog]: This master's current role is: LEADER
I20260619 04:36:16.638031 13305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0046ff439fa446828fcdc747e62956b3. Latest consensus state: current_term: 2 leader_uuid: "0046ff439fa446828fcdc747e62956b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0046ff439fa446828fcdc747e62956b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41715 } } }
I20260619 04:36:16.638301 13305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3 [sys.catalog]: This master's current role is: LEADER
I20260619 04:36:16.643181 13310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:36:16.649204 13310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:36:16.650705 13310 catalog_manager.cc:1259] Loaded cluster ID: a9b25b1691b849afab96bc4c96ae28fd
I20260619 04:36:16.650863 13310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:36:16.655041 13310 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:36:16.656591 13310 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3: Loaded TSK: 0
I20260619 04:36:16.658316 13310 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:36:16.661096 13299 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260619 04:36:17.004186 13301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:17.004688 13301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:17.004886 13301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:17.016171 13301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:17.016412 13301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:17.016697 13301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:36:17.032366 13301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:36311/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:37925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36653
--tserver_master_addrs=127.0.97.62:41715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:17.034242 13301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:17.036010 13301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:17.086995 13337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:17.091326 13336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:17.092958 13339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:17.093191 13301 server_base.cc:1034] running on GCE node
I20260619 04:36:17.094085 13301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:17.096024 13301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:17.097325 13301 hybrid_clock.cc:648] HybridClock initialized: now 1781843777097254 us; error 55 us; skew 500 ppm
I20260619 04:36:17.097772 13301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:17 dist-test-slave-n3tq krb5kdc[12253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:17.102227 13301 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:36:17.128906 13301 webserver.cc:460] Webserver started at http://127.0.97.1:36653/ using document root <none> and password file <none>
I20260619 04:36:17.130132 13301 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:17.130394 13301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 04:36:17 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 04:36:17.181555 13301 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.006s sys 0.002s
I20260619 04:36:17.189284 13348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:17.190847 13301 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.007s sys 0.000s
I20260619 04:36:17.191020 13301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "717b5617f19f463899b66ed4e343a864"
format_stamp: "Formatted at 2026-06-19 04:36:14 on dist-test-slave-n3tq"
server_key: "133419d2e4c788d54143238805986a70"
server_key_iv: "95a4482a4b53b1fa09abe5ba681fc0b2"
server_key_version: "kuduclusterkey@0"
I20260619 04:36:17.192247 13301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:17.213574 13301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:17.218014 13301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:17.218278 13301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:17.219411 13301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:36:17.220918 13301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:36:17.221012 13301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:17.221124 13301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:36:17.221195 13301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:17.275959 13301 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37925
I20260619 04:36:17.276039 13460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37925 every 8 connection(s)
I20260619 04:36:17.278113 13301 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:36:17.281087 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 13301
I20260619 04:36:17.281919 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:36311/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40391
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41715
--webserver_port=42689
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:36311/kms with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:36:17 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:36:17.326867 13464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:17.290099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41715 (local address 127.0.97.1:56553)
0619 04:36:17.291102 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:17.291165 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:17.293814 (+ 2649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:17.296332 (+ 2518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:17.296344 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:17.297331 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:17.298116 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:17.298144 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:17.300501 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:17.300506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:17.302399 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:17.302424 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:17.303829 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:17.303836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:17.304087 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:17.305313 (+ 1226us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:17.305343 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:17.307885 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:17.321359 (+ 13474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:17.321370 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:17.321389 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:17.322098 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:17.322669 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:17.322674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:17.322678 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:17.322933 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:17.324082 (+ 1149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:17.324091 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:17.324572 (+ 481us) client_negotiation.cc:769] Sending connection context
0619 04:36:17.324883 (+ 311us) client_negotiation.cc:240] Negotiation successful
0619 04:36:17.325455 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":1424,"threads_started":1}
I20260619 04:36:17.329245 13465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:17.291356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56553 (local address 127.0.97.62:41715)
0619 04:36:17.294658 (+ 3302us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:17.294679 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:17.294726 (+ 47us) server_negotiation.cc:407] Connection header received
0619 04:36:17.295183 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:17.295205 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:17.295703 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:17.296143 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:17.298377 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:17.300281 (+ 1904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:17.302566 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:17.303590 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:17.303786 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:17.308257 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:17.308279 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:17.308301 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:17.308352 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:17.321095 (+ 12743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:17.322297 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:17.322307 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:17.322318 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:17.322521 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:17.323070 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:17.323076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:17.323079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:17.323804 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:17.324005 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:17.327534 (+ 3529us) server_negotiation.cc:299] Negotiation successful
0619 04:36:17.327975 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3030,"thread_start_us":2814,"threads_started":1}
I20260619 04:36:17.333355 13461 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41715
I20260619 04:36:17.333654 13461 heartbeater.cc:464] Registering TS with master...
I20260619 04:36:17.334481 13461 heartbeater.cc:511] Master 127.0.97.62:41715 requested a full tablet report, sending...
W20260619 04:36:17.337570 13461 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41715 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 04:36:17.337774 13461 heartbeater.cc:464] Registering TS with master...
I20260619 04:36:17.338033 13461 heartbeater.cc:511] Master 127.0.97.62:41715 requested a full tablet report, sending...
I20260619 04:36:17.339246 13461 heartbeater.cc:464] Registering TS with master...
I20260619 04:36:17.339512 13461 heartbeater.cc:511] Master 127.0.97.62:41715 requested a full tablet report, sending...
W20260619 04:36:17.340590 13461 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 04:36:17.673094 13463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:17.673468 13463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:17.673651 13463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:17.682017 13463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:17.682150 13463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:17.682335 13463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:36:17.697827 13463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:36311/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=42689
--tserver_master_addrs=127.0.97.62:41715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:17.699688 13463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:17.701443 13463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:17.725495 13481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:17.725848 13480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:17.726086 13484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:17.734330 13463 server_base.cc:1034] running on GCE node
I20260619 04:36:17.735189 13463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:17.740295 13463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:17.741612 13463 hybrid_clock.cc:648] HybridClock initialized: now 1781843777741558 us; error 92 us; skew 500 ppm
I20260619 04:36:17.742060 13463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:17 dist-test-slave-n3tq krb5kdc[12253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:17.752414 13463 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:36:17.761963 13463 webserver.cc:460] Webserver started at http://127.0.97.2:42689/ using document root <none> and password file <none>
I20260619 04:36:17.762996 13463 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:17.763150 13463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 04:36:17 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 04:36:17.794165 13463 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.004s sys 0.000s
Jun 19 04:36:17 dist-test-slave-n3tq krb5kdc[12253](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781843777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:17.809217 13492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:17.811372 13463 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.001s sys 0.011s
I20260619 04:36:17.811640 13463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "f59e514c807d4105aca5434cbd5bfb64"
format_stamp: "Formatted at 2026-06-19 04:36:15 on dist-test-slave-n3tq"
server_key: "e032bcf62ca83602a983785dd6a7c69e"
server_key_iv: "928058b732b123c1dca064483c7cfbff"
server_key_version: "kuduclusterkey@0"
I20260619 04:36:17.813235 13463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:17.839990 13463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:17.845904 13463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:17.846314 13463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:17.847608 13463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:36:17.849474 13463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:36:17.849581 13463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:17.849677 13463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:36:17.849749 13463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:17.887894 13463 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40391
I20260619 04:36:17.887933 13605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40391 every 8 connection(s)
I20260619 04:36:17.889397 13463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:36:17.898125 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 13463
I20260619 04:36:17.898926 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:36311/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:38969
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41715
--webserver_port=34521
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:36311/kms with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:36:17 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:36:17.932818 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:17.893646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45471 (local address 127.0.97.62:41715)
0619 04:36:17.908920 (+ 15274us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:17.908926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:17.908951 (+ 25us) server_negotiation.cc:407] Connection header received
0619 04:36:17.911232 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:17.911238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:17.911329 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:17.911434 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:17.915989 (+ 4555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:17.917218 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:17.920027 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:17.920755 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:17.920859 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:17.925544 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:17.925551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:17.925556 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:17.925598 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:17.929092 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:17.930042 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:17.930048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:17.930052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:17.930214 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:17.930736 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:17.930741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:17.930745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:17.931088 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:17.931203 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:17.932184 (+ 981us) server_negotiation.cc:299] Negotiation successful
0619 04:36:17.932429 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15154,"thread_start_us":15311,"threads_started":1}
I20260619 04:36:17.934726 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:17.893012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41715 (local address 127.0.97.2:45471)
0619 04:36:17.908210 (+ 15198us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:17.908277 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:17.910814 (+ 2537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:17.914226 (+ 3412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:17.914240 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:17.914879 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:17.915741 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:17.915770 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:17.917458 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:17.917464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:17.919806 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:17.919821 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:17.921228 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:17.921235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:17.921548 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:17.922546 (+ 998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:17.922577 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:17.925255 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:17.929265 (+ 4010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:17.929275 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:17.929293 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:17.929904 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:17.930366 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:17.930371 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:17.930375 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:17.930607 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:17.931299 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:17.931308 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:17.931748 (+ 440us) client_negotiation.cc:769] Sending connection context
0619 04:36:17.933143 (+ 1395us) client_negotiation.cc:240] Negotiation successful
0619 04:36:17.933721 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14799,"thread_start_us":18117,"threads_started":1}
I20260619 04:36:17.936269 13606 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41715
I20260619 04:36:17.936619 13606 heartbeater.cc:464] Registering TS with master...
I20260619 04:36:17.937316 13606 heartbeater.cc:511] Master 127.0.97.62:41715 requested a full tablet report, sending...
I20260619 04:36:17.939121 13264 ts_manager.cc:194] Registered new tserver with Master: f59e514c807d4105aca5434cbd5bfb64 (127.0.97.2:40391)
I20260619 04:36:17.940918 13264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45471
Jun 19 04:36:18 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 04:36:18.340732 13461 heartbeater.cc:464] Registering TS with master...
I20260619 04:36:18.341147 13461 heartbeater.cc:511] Master 127.0.97.62:41715 requested a full tablet report, sending...
I20260619 04:36:18.342375 13264 ts_manager.cc:194] Registered new tserver with Master: 717b5617f19f463899b66ed4e343a864 (127.0.97.1:37925)
I20260619 04:36:18.343575 13264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56553
W20260619 04:36:18.389487 13610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:18.389895 13610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:18.390105 13610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:18.401633 13610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:18.401787 13610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:18.402032 13610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:36:18.417697 13610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:36311/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:38969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34521
--tserver_master_addrs=127.0.97.62:41715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:18.419610 13610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:18.421478 13610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:18.463924 13615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:18.470767 13616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:18.476748 13610 server_base.cc:1034] running on GCE node
W20260619 04:36:18.477243 13619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:18.478082 13610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:18.496747 13610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:18.498096 13610 hybrid_clock.cc:648] HybridClock initialized: now 1781843778498018 us; error 74 us; skew 500 ppm
I20260619 04:36:18.499418 13610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:18 dist-test-slave-n3tq krb5kdc[12253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:18.504396 13610 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:36:18.525265 13610 webserver.cc:460] Webserver started at http://127.0.97.3:34521/ using document root <none> and password file <none>
I20260619 04:36:18.526278 13610 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:18.526505 13610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 04:36:18 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 04:36:18.565413 13610 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.004s sys 0.000s
I20260619 04:36:18.575162 13626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:18.577026 13610 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.006s sys 0.003s
I20260619 04:36:18.577288 13610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "40d53f48115948f9ab52975a81406d60"
format_stamp: "Formatted at 2026-06-19 04:36:15 on dist-test-slave-n3tq"
server_key: "74f8b336d978210f94b6efc868daa9e0"
server_key_iv: "c41b7028a2abf2ee7845eefa617d9b4e"
server_key_version: "kuduclusterkey@0"
I20260619 04:36:18.578725 13610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:18.654595 13610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:18.672916 13610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:18.673264 13610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:18.674433 13610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:36:18.677253 13610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:36:18.677400 13610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260619 04:36:18.677536 13610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:36:18.677645 13610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 04:36:18 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 04:36:18.897181 13610 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38969
I20260619 04:36:18.897241 13739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38969 every 8 connection(s)
I20260619 04:36:18.899080 13610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:36:18.902954 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 13610
Jun 19 04:36:18 dist-test-slave-n3tq krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:36:18.938452 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:18.904304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43787 (local address 127.0.97.62:41715)
0619 04:36:18.905113 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:18.905120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:18.909743 (+ 4623us) server_negotiation.cc:407] Connection header received
0619 04:36:18.912744 (+ 3001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:18.912750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:18.912834 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:18.912934 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:18.920186 (+ 7252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:18.921393 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:18.923916 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:18.924594 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:18.924696 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:18.930434 (+ 5738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:18.930441 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:18.930446 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:18.930504 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:18.934067 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:18.935021 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:18.935026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:18.935030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:18.935168 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:18.935807 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:18.935813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:18.935817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:18.936298 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:18.936608 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:18.937799 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0619 04:36:18.938041 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":4300,"threads_started":1}
I20260619 04:36:18.940764 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:18.903590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41715 (local address 127.0.97.3:43787)
0619 04:36:18.909553 (+ 5963us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:18.909618 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:18.912390 (+ 2772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:18.918302 (+ 5912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:18.918322 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:18.919095 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:18.919980 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:18.920007 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:18.921581 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:18.921587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:18.923730 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:18.923745 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:18.925110 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:18.925117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:18.925396 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:18.926212 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:18.926241 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:18.928881 (+ 2640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:18.934235 (+ 5354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:18.934245 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:18.934265 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:18.934884 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:18.935368 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:18.935373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:18.935376 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:18.935623 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:18.936746 (+ 1123us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:18.936762 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:18.937344 (+ 582us) client_negotiation.cc:769] Sending connection context
0619 04:36:18.938678 (+ 1334us) client_negotiation.cc:240] Negotiation successful
0619 04:36:18.939281 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5591,"thread_start_us":5567,"threads_started":1}
I20260619 04:36:18.943048 13740 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41715
I20260619 04:36:18.943398 13740 heartbeater.cc:464] Registering TS with master...
I20260619 04:36:18.944308 13740 heartbeater.cc:511] Master 127.0.97.62:41715 requested a full tablet report, sending...
I20260619 04:36:18.944585 13606 heartbeater.cc:503] Master 127.0.97.62:41715 was elected leader, sending a full tablet report...
I20260619 04:36:18.946069 13263 ts_manager.cc:194] Registered new tserver with Master: 40d53f48115948f9ab52975a81406d60 (127.0.97.3:38969)
I20260619 04:36:18.947281 13263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43787
I20260619 04:36:18.948958 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:18.904616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.0.97.62:41715)
0619 04:36:18.908902 (+ 4286us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:18.908909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:18.918347 (+ 9438us) server_negotiation.cc:407] Connection header received
0619 04:36:18.918426 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:18.918431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:18.918518 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:18.918613 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:18.926088 (+ 7475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:18.927279 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:18.930276 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:18.931019 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:18.931140 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:18.939357 (+ 8217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:18.939363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:18.939368 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:18.939401 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:18.942667 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:18.943984 (+ 1317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:18.943989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:18.943992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:18.944127 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:18.944598 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:18.944602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:18.944605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:18.944936 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:18.945669 (+ 733us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:18.948309 (+ 2640us) server_negotiation.cc:299] Negotiation successful
0619 04:36:18.948580 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4141,"thread_start_us":642,"threads_started":1}
I20260619 04:36:18.952232 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:36:18.969506 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:18.955166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33362 (local address 127.0.97.62:41715)
0619 04:36:18.955384 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:18.955390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:18.955714 (+ 324us) server_negotiation.cc:407] Connection header received
0619 04:36:18.955876 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:18.955879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:18.955950 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:18.956028 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:18.957906 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:18.958598 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:18.960564 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:18.960986 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:18.961086 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:18.964215 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:18.964220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:18.964223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:18.964255 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:18.966745 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:18.967564 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:18.967569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:18.967572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:18.967703 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:18.968117 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:18.968121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:18.968124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:18.968501 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:18.968704 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:18.969083 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 04:36:18.969256 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 04:36:18.974654 13263 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:36:18.982000 13263 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:36:19.033345 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:18.998353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56230 (local address 127.0.97.1:37925)
0619 04:36:19.023116 (+ 24763us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:19.023124 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:19.023159 (+ 35us) server_negotiation.cc:407] Connection header received
0619 04:36:19.023236 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:19.023242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:19.023502 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:19.023728 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:36:19.024412 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.025897 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.029442 (+ 3545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.030865 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.030982 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:19.031315 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:19.032815 (+ 1500us) server_negotiation.cc:299] Negotiation successful
0619 04:36:19.032998 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24603,"thread_start_us":25568,"threads_started":1}
I20260619 04:36:19.034204 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:18.997330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38969 (local address 127.0.0.1:49996)
0619 04:36:19.020791 (+ 23461us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:19.020819 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:19.021007 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:19.025880 (+ 4873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:19.025884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:19.025919 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:36:19.026392 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:19.026403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.028233 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.028237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:19.029974 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:19.029985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.031959 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.031966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:19.032383 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:19.032485 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 04:36:19.032618 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 04:36:19.033786 (+ 1168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23278,"mutex_wait_us":270,"thread_start_us":24088,"threads_started":1}
I20260619 04:36:19.035135 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:18.998051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37925 (local address 127.0.0.1:56230)
0619 04:36:19.021678 (+ 23627us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:19.021699 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:19.021807 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:19.023767 (+ 1960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:19.023774 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:19.023806 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:36:19.024195 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:19.024211 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.026102 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.026106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:19.029273 (+ 3167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:19.029287 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.031501 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.031508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:19.032389 (+ 881us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:19.032483 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 04:36:19.032619 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 04:36:19.034793 (+ 2174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23499,"mutex_wait_us":704,"thread_start_us":23064,"threads_started":1}
I20260619 04:36:19.035117 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:18.997946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49996 (local address 127.0.97.3:38969)
0619 04:36:19.024616 (+ 26670us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:19.024624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:19.024678 (+ 54us) server_negotiation.cc:407] Connection header received
0619 04:36:19.024768 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:19.024775 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:19.025024 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:19.025225 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:36:19.026584 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.028124 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.030110 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.031464 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.031585 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:19.031819 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:19.034597 (+ 2778us) server_negotiation.cc:299] Negotiation successful
0619 04:36:19.034772 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26475,"thread_start_us":26318,"threads_started":1}
I20260619 04:36:19.036681 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:18.996402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40391 (local address 127.0.0.1:45574)
0619 04:36:18.997258 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:18.997300 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:18.997489 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:19.026601 (+ 29112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:19.026606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:19.026635 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:36:19.027021 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:19.027034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.030975 (+ 3941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.030981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:19.034081 (+ 3100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:19.034095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.035942 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.035948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:19.036106 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:19.036139 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 04:36:19.036198 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 04:36:19.036327 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":123,"thread_start_us":819,"threads_started":1}
I20260619 04:36:19.037328 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:18.997038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45574 (local address 127.0.97.2:40391)
0619 04:36:19.023061 (+ 26023us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:19.023070 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:19.023103 (+ 33us) server_negotiation.cc:407] Connection header received
0619 04:36:19.023212 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:19.023221 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:19.023505 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:19.027222 (+ 3717us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:36:19.027469 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.030828 (+ 3359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.034283 (+ 3455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.035733 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.036582 (+ 849us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:19.036810 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:19.036882 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 04:36:19.037034 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25837,"thread_start_us":29079,"threads_started":1}
I20260619 04:36:19.039112 13675 tablet_service.cc:1467] Processing CreateTablet for tablet 1e25d4cbbf054f91b4ffa24c9f55bfcb (DEFAULT_TABLE table=test-table [id=9b90b858f8c8417a8c291045559d9978]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:36:19.039135 13396 tablet_service.cc:1467] Processing CreateTablet for tablet 1e25d4cbbf054f91b4ffa24c9f55bfcb (DEFAULT_TABLE table=test-table [id=9b90b858f8c8417a8c291045559d9978]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:36:19.040330 13675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e25d4cbbf054f91b4ffa24c9f55bfcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:36:19.040555 13396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e25d4cbbf054f91b4ffa24c9f55bfcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:36:19.041085 13540 tablet_service.cc:1467] Processing CreateTablet for tablet 1e25d4cbbf054f91b4ffa24c9f55bfcb (DEFAULT_TABLE table=test-table [id=9b90b858f8c8417a8c291045559d9978]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:36:19.042433 13540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e25d4cbbf054f91b4ffa24c9f55bfcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:36:19.058183 13769 tablet_bootstrap.cc:492] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864: Bootstrap starting.
I20260619 04:36:19.064312 13769 tablet_bootstrap.cc:654] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864: Neither blocks nor log segments found. Creating new log.
I20260619 04:36:19.065240 13768 tablet_bootstrap.cc:492] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 40d53f48115948f9ab52975a81406d60: Bootstrap starting.
I20260619 04:36:19.066684 13770 tablet_bootstrap.cc:492] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P f59e514c807d4105aca5434cbd5bfb64: Bootstrap starting.
I20260619 04:36:19.068873 13768 tablet_bootstrap.cc:654] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 40d53f48115948f9ab52975a81406d60: Neither blocks nor log segments found. Creating new log.
I20260619 04:36:19.069865 13769 log.cc:824] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864: Log is configured to *not* fsync() on all Append() calls
I20260619 04:36:19.070042 13768 log.cc:824] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 40d53f48115948f9ab52975a81406d60: Log is configured to *not* fsync() on all Append() calls
I20260619 04:36:19.070117 13770 tablet_bootstrap.cc:654] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P f59e514c807d4105aca5434cbd5bfb64: Neither blocks nor log segments found. Creating new log.
I20260619 04:36:19.072443 13770 log.cc:824] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P f59e514c807d4105aca5434cbd5bfb64: Log is configured to *not* fsync() on all Append() calls
I20260619 04:36:19.076292 13768 tablet_bootstrap.cc:492] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 40d53f48115948f9ab52975a81406d60: No bootstrap required, opened a new log
I20260619 04:36:19.076683 13768 ts_tablet_manager.cc:1397] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 40d53f48115948f9ab52975a81406d60: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260619 04:36:19.087512 13769 tablet_bootstrap.cc:492] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864: No bootstrap required, opened a new log
I20260619 04:36:19.087896 13769 ts_tablet_manager.cc:1397] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.009s
I20260619 04:36:19.088151 13770 tablet_bootstrap.cc:492] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P f59e514c807d4105aca5434cbd5bfb64: No bootstrap required, opened a new log
I20260619 04:36:19.088531 13770 ts_tablet_manager.cc:1397] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P f59e514c807d4105aca5434cbd5bfb64: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.004s
I20260619 04:36:19.088990 13768 raft_consensus.cc:348] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 40d53f48115948f9ab52975a81406d60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59e514c807d4105aca5434cbd5bfb64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "40d53f48115948f9ab52975a81406d60" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38969 } } peers { permanent_uuid: "717b5617f19f463899b66ed4e343a864" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37925 } }
I20260619 04:36:19.089460 13768 raft_consensus.cc:374] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 40d53f48115948f9ab52975a81406d60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:36:19.089578 13768 raft_consensus.cc:729] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 40d53f48115948f9ab52975a81406d60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d53f48115948f9ab52975a81406d60, State: Initialized, Role: FOLLOWER
I20260619 04:36:19.090281 13768 consensus_queue.cc:260] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 40d53f48115948f9ab52975a81406d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59e514c807d4105aca5434cbd5bfb64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "40d53f48115948f9ab52975a81406d60" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38969 } } peers { permanent_uuid: "717b5617f19f463899b66ed4e343a864" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37925 } }
I20260619 04:36:19.098763 13769 raft_consensus.cc:348] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59e514c807d4105aca5434cbd5bfb64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "40d53f48115948f9ab52975a81406d60" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38969 } } peers { permanent_uuid: "717b5617f19f463899b66ed4e343a864" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37925 } }
I20260619 04:36:19.099007 13770 raft_consensus.cc:348] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P f59e514c807d4105aca5434cbd5bfb64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59e514c807d4105aca5434cbd5bfb64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "40d53f48115948f9ab52975a81406d60" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38969 } } peers { permanent_uuid: "717b5617f19f463899b66ed4e343a864" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37925 } }
I20260619 04:36:19.099313 13769 raft_consensus.cc:374] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:36:19.099406 13769 raft_consensus.cc:729] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717b5617f19f463899b66ed4e343a864, State: Initialized, Role: FOLLOWER
I20260619 04:36:19.099529 13770 raft_consensus.cc:374] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P f59e514c807d4105aca5434cbd5bfb64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:36:19.099653 13770 raft_consensus.cc:729] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P f59e514c807d4105aca5434cbd5bfb64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f59e514c807d4105aca5434cbd5bfb64, State: Initialized, Role: FOLLOWER
I20260619 04:36:19.100636 13770 consensus_queue.cc:260] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P f59e514c807d4105aca5434cbd5bfb64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59e514c807d4105aca5434cbd5bfb64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "40d53f48115948f9ab52975a81406d60" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38969 } } peers { permanent_uuid: "717b5617f19f463899b66ed4e343a864" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37925 } }
I20260619 04:36:19.101716 13740 heartbeater.cc:503] Master 127.0.97.62:41715 was elected leader, sending a full tablet report...
I20260619 04:36:19.101895 13769 consensus_queue.cc:260] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59e514c807d4105aca5434cbd5bfb64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "40d53f48115948f9ab52975a81406d60" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38969 } } peers { permanent_uuid: "717b5617f19f463899b66ed4e343a864" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37925 } }
I20260619 04:36:19.102406 13768 ts_tablet_manager.cc:1428] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 40d53f48115948f9ab52975a81406d60: Time spent starting tablet: real 0.026s user 0.016s sys 0.008s
I20260619 04:36:19.103654 13461 heartbeater.cc:503] Master 127.0.97.62:41715 was elected leader, sending a full tablet report...
I20260619 04:36:19.103957 13769 ts_tablet_manager.cc:1428] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864: Time spent starting tablet: real 0.016s user 0.004s sys 0.009s
I20260619 04:36:19.105130 13770 ts_tablet_manager.cc:1428] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P f59e514c807d4105aca5434cbd5bfb64: Time spent starting tablet: real 0.016s user 0.012s sys 0.000s
I20260619 04:36:19.122331 13775 raft_consensus.cc:482] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:36:19.122632 13775 raft_consensus.cc:504] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59e514c807d4105aca5434cbd5bfb64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "40d53f48115948f9ab52975a81406d60" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38969 } } peers { permanent_uuid: "717b5617f19f463899b66ed4e343a864" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37925 } }
I20260619 04:36:19.124686 13775 leader_election.cc:288] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f59e514c807d4105aca5434cbd5bfb64 (127.0.97.2:40391), 40d53f48115948f9ab52975a81406d60 (127.0.97.3:38969)
I20260619 04:36:19.132236 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:19.125204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45889 (local address 127.0.97.2:40391)
0619 04:36:19.125392 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:19.125398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:19.125988 (+ 590us) server_negotiation.cc:407] Connection header received
0619 04:36:19.126299 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:19.126305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:19.126398 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:19.126478 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:36:19.127192 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.128232 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.130484 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.131285 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.131402 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:19.131502 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:19.131856 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 04:36:19.131989 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 04:36:19.132337 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:19.124876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40391 (local address 127.0.97.1:45889)
0619 04:36:19.125886 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:19.125904 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:19.126114 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:19.126521 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:19.126525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:19.126547 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:36:19.127025 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:19.127038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.128358 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.128363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:19.130353 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:19.130364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.131478 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.131484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:19.131642 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:19.131684 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 04:36:19.131820 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 04:36:19.131952 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":2,"spinlock_wait_cycles":48896,"thread_start_us":679,"threads_started":1}
I20260619 04:36:19.132439 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:19.125112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52729 (local address 127.0.97.3:38969)
0619 04:36:19.125360 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:19.125367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:19.126353 (+ 986us) server_negotiation.cc:407] Connection header received
0619 04:36:19.126495 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:19.126500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:19.126569 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:19.126650 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:36:19.127276 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.127956 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.130397 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.131552 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.131673 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:19.131768 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:19.132085 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 04:36:19.132227 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260619 04:36:19.133045 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:19.124876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38969 (local address 127.0.97.1:52729)
0619 04:36:19.126242 (+ 1366us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:19.126258 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:19.126382 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:19.126666 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:19.126671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:19.126706 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:36:19.127125 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:19.127134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.128082 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.128087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:19.130221 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:19.130238 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.131754 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.131761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:19.131919 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:19.131946 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 04:36:19.132722 (+ 776us) client_negotiation.cc:240] Negotiation successful
0619 04:36:19.132814 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1251,"mutex_wait_us":527,"thread_start_us":541,"threads_started":1}
I20260619 04:36:19.133499 13560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e25d4cbbf054f91b4ffa24c9f55bfcb" candidate_uuid: "717b5617f19f463899b66ed4e343a864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f59e514c807d4105aca5434cbd5bfb64" is_pre_election: true
I20260619 04:36:19.133864 13695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e25d4cbbf054f91b4ffa24c9f55bfcb" candidate_uuid: "717b5617f19f463899b66ed4e343a864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40d53f48115948f9ab52975a81406d60" is_pre_election: true
I20260619 04:36:19.134079 13560 raft_consensus.cc:2454] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P f59e514c807d4105aca5434cbd5bfb64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717b5617f19f463899b66ed4e343a864 in term 0.
I20260619 04:36:19.134334 13695 raft_consensus.cc:2454] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 40d53f48115948f9ab52975a81406d60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717b5617f19f463899b66ed4e343a864 in term 0.
I20260619 04:36:19.134788 13352 leader_election.cc:302] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717b5617f19f463899b66ed4e343a864, f59e514c807d4105aca5434cbd5bfb64; no voters:
I20260619 04:36:19.135177 13775 raft_consensus.cc:2781] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:36:19.135305 13775 raft_consensus.cc:482] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:36:19.135401 13775 raft_consensus.cc:3037] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:36:19.137617 13775 raft_consensus.cc:504] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59e514c807d4105aca5434cbd5bfb64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "40d53f48115948f9ab52975a81406d60" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38969 } } peers { permanent_uuid: "717b5617f19f463899b66ed4e343a864" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37925 } }
I20260619 04:36:19.138206 13775 leader_election.cc:288] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864 [CANDIDATE]: Term 1 election: Requested vote from peers f59e514c807d4105aca5434cbd5bfb64 (127.0.97.2:40391), 40d53f48115948f9ab52975a81406d60 (127.0.97.3:38969)
I20260619 04:36:19.138643 13695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e25d4cbbf054f91b4ffa24c9f55bfcb" candidate_uuid: "717b5617f19f463899b66ed4e343a864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40d53f48115948f9ab52975a81406d60"
I20260619 04:36:19.138654 13560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e25d4cbbf054f91b4ffa24c9f55bfcb" candidate_uuid: "717b5617f19f463899b66ed4e343a864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f59e514c807d4105aca5434cbd5bfb64"
I20260619 04:36:19.138855 13695 raft_consensus.cc:3037] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 40d53f48115948f9ab52975a81406d60 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:36:19.138855 13560 raft_consensus.cc:3037] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P f59e514c807d4105aca5434cbd5bfb64 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:36:19.141954 13560 raft_consensus.cc:2454] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P f59e514c807d4105aca5434cbd5bfb64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717b5617f19f463899b66ed4e343a864 in term 1.
I20260619 04:36:19.141954 13695 raft_consensus.cc:2454] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 40d53f48115948f9ab52975a81406d60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717b5617f19f463899b66ed4e343a864 in term 1.
I20260619 04:36:19.142373 13351 leader_election.cc:302] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40d53f48115948f9ab52975a81406d60, 717b5617f19f463899b66ed4e343a864; no voters:
I20260619 04:36:19.142719 13775 raft_consensus.cc:2781] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:36:19.143177 13775 raft_consensus.cc:686] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864 [term 1 LEADER]: Becoming Leader. State: Replica: 717b5617f19f463899b66ed4e343a864, State: Running, Role: LEADER
I20260619 04:36:19.143678 13775 consensus_queue.cc:237] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59e514c807d4105aca5434cbd5bfb64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "40d53f48115948f9ab52975a81406d60" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38969 } } peers { permanent_uuid: "717b5617f19f463899b66ed4e343a864" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37925 } }
I20260619 04:36:19.148593 13263 catalog_manager.cc:5495] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864 reported cstate change: term changed from 0 to 1, leader changed from <none> to 717b5617f19f463899b66ed4e343a864 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "717b5617f19f463899b66ed4e343a864" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59e514c807d4105aca5434cbd5bfb64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40d53f48115948f9ab52975a81406d60" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "717b5617f19f463899b66ed4e343a864" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37925 } health_report { overall_health: HEALTHY } } }
W20260619 04:36:19.161085 13607 tablet.cc:2369] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P f59e514c807d4105aca5434cbd5bfb64: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:36:19.171234 13741 tablet.cc:2369] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 40d53f48115948f9ab52975a81406d60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:36:19.188036 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:19.182100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56232 (local address 127.0.97.1:37925)
0619 04:36:19.182325 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:19.182332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:19.182353 (+ 21us) server_negotiation.cc:407] Connection header received
0619 04:36:19.182455 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:19.182460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:19.182570 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:19.182719 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:36:19.183048 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.184084 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.185575 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.186133 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.186264 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:19.186728 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:36:19.187428 (+ 700us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:36:19.187575 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:19.187678 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 04:36:19.187764 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 04:36:19.196241 13695 raft_consensus.cc:1261] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 40d53f48115948f9ab52975a81406d60 [term 1 FOLLOWER]: Refusing update from remote peer 717b5617f19f463899b66ed4e343a864: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:36:19.196383 13560 raft_consensus.cc:1261] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P f59e514c807d4105aca5434cbd5bfb64 [term 1 FOLLOWER]: Refusing update from remote peer 717b5617f19f463899b66ed4e343a864: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:36:19.197266 13775 consensus_queue.cc:1035] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40d53f48115948f9ab52975a81406d60" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:36:19.197621 13779 consensus_queue.cc:1035] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f59e514c807d4105aca5434cbd5bfb64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:36:19.209724 13783 mvcc.cc:205] Tried to move back new op lower bound from 7298432119577837568 to 7298432119377076224. Current Snapshot: MvccSnapshot[applied={T|T < 7298432119577837568}]
I20260619 04:36:19.212957 13782 mvcc.cc:205] Tried to move back new op lower bound from 7298432119577837568 to 7298432119377076224. Current Snapshot: MvccSnapshot[applied={T|T < 7298432119577837568 or (T in {7298432119577837568})}]
I20260619 04:36:19.227130 13263 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:36:19.227413 13263 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:36:19.231843 13263 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0046ff439fa446828fcdc747e62956b3: Sending DeleteTablet for 3 replicas of tablet 1e25d4cbbf054f91b4ffa24c9f55bfcb
I20260619 04:36:19.233217 13540 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e25d4cbbf054f91b4ffa24c9f55bfcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:36:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45574
I20260619 04:36:19.235492 13675 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e25d4cbbf054f91b4ffa24c9f55bfcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:36:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49996
I20260619 04:36:19.236209 13396 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e25d4cbbf054f91b4ffa24c9f55bfcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:36:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56230
I20260619 04:36:19.245062 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 13301
I20260619 04:36:19.255015 13792 tablet_replica.cc:330] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 40d53f48115948f9ab52975a81406d60: stopping tablet replica
I20260619 04:36:19.278939 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:19.249253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56242 (local address 127.0.97.1:37925)
0619 04:36:19.249420 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:19.249426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:19.249443 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:36:19.249503 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:19.249508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:19.249577 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:19.251455 (+ 1878us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:19.251603 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.252800 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.254816 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.255510 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.255621 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:19.259523 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:19.259541 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:19.259566 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:19.259619 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:19.267679 (+ 8060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:19.273173 (+ 5494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:19.273183 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:19.273186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:19.273365 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:19.274596 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:19.274602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:19.274605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:19.275091 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:19.278181 (+ 3090us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:19.278333 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 04:36:19.278578 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 04:36:19.281358 13456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:19.280615 13793 tablet_replica.cc:330] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864: stopping tablet replica
I20260619 04:36:19.290052 13791 tablet_replica.cc:330] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P f59e514c807d4105aca5434cbd5bfb64: stopping tablet replica
I20260619 04:36:19.290482 13791 raft_consensus.cc:2229] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P f59e514c807d4105aca5434cbd5bfb64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:36:19.291006 13791 raft_consensus.cc:2258] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P f59e514c807d4105aca5434cbd5bfb64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:36:19.293582 13791 ts_tablet_manager.cc:1905] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P f59e514c807d4105aca5434cbd5bfb64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:36:19.299045 13792 raft_consensus.cc:2229] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 40d53f48115948f9ab52975a81406d60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:36:19.299966 13792 raft_consensus.cc:2258] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 40d53f48115948f9ab52975a81406d60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:36:19.300917 13791 ts_tablet_manager.cc:1918] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P f59e514c807d4105aca5434cbd5bfb64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:36:19.301059 13791 log.cc:1196] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P f59e514c807d4105aca5434cbd5bfb64: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/1e25d4cbbf054f91b4ffa24c9f55bfcb
I20260619 04:36:19.301638 13791 ts_tablet_manager.cc:1939] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P f59e514c807d4105aca5434cbd5bfb64: Deleting consensus metadata
I20260619 04:36:19.303560 13251 catalog_manager.cc:4841] TS f59e514c807d4105aca5434cbd5bfb64 (127.0.97.2:40391): tablet 1e25d4cbbf054f91b4ffa24c9f55bfcb (table test-table [id=9b90b858f8c8417a8c291045559d9978]) successfully deleted
I20260619 04:36:19.306193 13792 ts_tablet_manager.cc:1905] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 40d53f48115948f9ab52975a81406d60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:36:19.312619 13792 ts_tablet_manager.cc:1918] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 40d53f48115948f9ab52975a81406d60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:36:19.312780 13792 log.cc:1196] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 40d53f48115948f9ab52975a81406d60: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/1e25d4cbbf054f91b4ffa24c9f55bfcb
I20260619 04:36:19.313230 13792 ts_tablet_manager.cc:1939] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 40d53f48115948f9ab52975a81406d60: Deleting consensus metadata
I20260619 04:36:19.314962 13250 catalog_manager.cc:4841] TS 40d53f48115948f9ab52975a81406d60 (127.0.97.3:38969): tablet 1e25d4cbbf054f91b4ffa24c9f55bfcb (table test-table [id=9b90b858f8c8417a8c291045559d9978]) successfully deleted
I20260619 04:36:19.289256 13793 raft_consensus.cc:2229] T 1e25d4cbbf054f91b4ffa24c9f55bfcb P 717b5617f19f463899b66ed4e343a864 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:36:19.402529 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 13301
I20260619 04:36:19.416355 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 13463
W20260619 04:36:19.416442 13251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:37925 (error 108)
I20260619 04:36:19.418219 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:19.417831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37925 (local address 127.0.0.1:56254)
0619 04:36:19.417990 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:19.418092 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 04:36:19.418484 13251 catalog_manager.cc:4584] TS 717b5617f19f463899b66ed4e343a864 (127.0.97.1:37925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e25d4cbbf054f91b4ffa24c9f55bfcb: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37925: connect: Connection refused (error 111)
I20260619 04:36:19.433908 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:19.417145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45588 (local address 127.0.97.2:40391)
0619 04:36:19.417348 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:19.417355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:19.417418 (+ 63us) server_negotiation.cc:407] Connection header received
0619 04:36:19.417592 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:19.417597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:19.417689 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:19.417781 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:19.419191 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.419966 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.421204 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.421608 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.421720 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:19.427916 (+ 6196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:19.427930 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:19.427950 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:19.427993 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:19.431437 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:19.432115 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:19.432122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:19.432124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:19.432263 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:19.432636 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:19.432639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:19.432641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:19.432991 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:19.433217 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:19.433407 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 04:36:19.433568 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 04:36:19.434549 13600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:19.472133 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:19.471638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37925 (local address 127.0.0.1:56264)
0619 04:36:19.471854 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:19.471986 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 04:36:19.473155 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:19.472849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37925 (local address 127.0.0.1:56278)
0619 04:36:19.472964 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:19.473033 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 04:36:19.521907 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 13463
I20260619 04:36:19.533578 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:19.533002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37925 (local address 127.0.0.1:56282)
0619 04:36:19.533258 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:19.533367 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 04:36:19.534804 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:19.534387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37925 (local address 127.0.0.1:56292)
0619 04:36:19.534543 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:19.534641 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 04:36:19.535656 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 13610
I20260619 04:36:19.547706 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:19.536303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50000 (local address 127.0.97.3:38969)
0619 04:36:19.536512 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:19.536516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:19.536534 (+ 18us) server_negotiation.cc:407] Connection header received
0619 04:36:19.536652 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:19.536656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:19.536730 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:19.536810 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:19.537895 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.538589 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.539754 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:19.540160 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:19.540285 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:19.542705 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:19.542721 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:19.542737 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:19.542771 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:19.545627 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:19.546217 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:19.546224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:19.546226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:19.546335 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:19.546600 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:19.546603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:19.546604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:19.546932 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:19.547133 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:19.547320 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 04:36:19.547470 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 04:36:19.548370 13735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:19.615931 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:19.615447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37925 (local address 127.0.0.1:56302)
0619 04:36:19.615662 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:19.615757 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 04:36:19.616948 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:19.616575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37925 (local address 127.0.0.1:56306)
0619 04:36:19.616761 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:19.616831 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 04:36:19.634618 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 13610
I20260619 04:36:19.647286 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 13231
I20260619 04:36:19.648401 13294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:19.758415 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 13231
I20260619 04:36:19.776298 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 04:36:20.110486 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 04:36:20.110692 388 mini_ranger.cc:66] Stopping Ranger...
I20260619 04:36:20.455742 388 mini_ranger.cc:68] Stopped Ranger
2026-06-19 04:36:20.455 UTC [12283] LOG: received smart shutdown request
2026-06-19 04:36:20.458 UTC [12283] LOG: background worker "logical replication launcher" (PID 12292) exited with exit code 1
2026-06-19 04:36:20.458 UTC [12287] LOG: shutting down
2026-06-19 04:36:20.583 UTC [12283] LOG: database system is shut down
2026-06-19T04:36:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49405 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:36:20 dist-test-slave-n3tq krb5kdc[13807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:36:20 dist-test-slave-n3tq krb5kdc[13807](info): set up 2 sockets
Jun 19 04:36:20 dist-test-slave-n3tq krb5kdc[13807](info): commencing operation
krb5kdc: starting...
W20260619 04:36:21.688374 388 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:36:21 dist-test-slave-n3tq krb5kdc[13807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:36:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:36:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:36:21.887032 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44377
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44377,127.0.97.61:44327,127.0.97.60:35879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:36:22.197642 13820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:22.197952 13820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:22.198086 13820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:22.205528 13820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:36:22.205641 13820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:22.205715 13820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:22.205787 13820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:36:22.205842 13820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:36:22.215826 13820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44377,127.0.97.61:44327,127.0.97.60:35879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:22.217146 13820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:22.218396 13820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:22.229630 13826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:22.229605 13828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:22.229817 13825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:22.230369 13820 server_base.cc:1034] running on GCE node
I20260619 04:36:22.231128 13820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:22.232532 13820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:22.233732 13820 hybrid_clock.cc:648] HybridClock initialized: now 1781843782233682 us; error 45 us; skew 500 ppm
I20260619 04:36:22.234081 13820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:22 dist-test-slave-n3tq krb5kdc[13807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:22.236965 13820 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:36:22.238317 13820 webserver.cc:460] Webserver started at http://127.0.97.62:37527/ using document root <none> and password file <none>
I20260619 04:36:22.238969 13820 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:22.239079 13820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:22.239370 13820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:36:22.242365 13820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "2299da07d0814549b99bf5a6a41bd6b6"
format_stamp: "Formatted at 2026-06-19 04:36:22 on dist-test-slave-n3tq"
server_key: "1887b61180e2b944e5d652fbd6e505cc"
server_key_iv: "c83820470996678a365b3d88e1765850"
server_key_version: "encryptionkey@0"
I20260619 04:36:22.243101 13820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "2299da07d0814549b99bf5a6a41bd6b6"
format_stamp: "Formatted at 2026-06-19 04:36:22 on dist-test-slave-n3tq"
server_key: "1887b61180e2b944e5d652fbd6e505cc"
server_key_iv: "c83820470996678a365b3d88e1765850"
server_key_version: "encryptionkey@0"
I20260619 04:36:22.247972 13820 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:36:22.251201 13835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:22.252504 13820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:36:22.252655 13820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "2299da07d0814549b99bf5a6a41bd6b6"
format_stamp: "Formatted at 2026-06-19 04:36:22 on dist-test-slave-n3tq"
server_key: "1887b61180e2b944e5d652fbd6e505cc"
server_key_iv: "c83820470996678a365b3d88e1765850"
server_key_version: "encryptionkey@0"
I20260619 04:36:22.252826 13820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:22.274030 13820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:22.278209 13820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:22.278470 13820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:22.295692 13820 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44377
I20260619 04:36:22.295725 13886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44377 every 8 connection(s)
I20260619 04:36:22.297075 13820 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:36:22.300685 13887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:36:22.305362 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 13820
I20260619 04:36:22.305589 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:36:22.306362 388 external_mini_cluster.cc:1428] Setting key 32ad9c3baac8936ecffc78d1fccf2fe6
I20260619 04:36:22.307881 13887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:36:22 dist-test-slave-n3tq krb5kdc[13807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260619 04:36:22.330158 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.311516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40744 (local address 127.0.97.62:44377)
0619 04:36:22.313223 (+ 1707us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:22.313232 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:22.313282 (+ 50us) server_negotiation.cc:407] Connection header received
0619 04:36:22.314755 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:22.314760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:22.314977 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:22.315161 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:22.316411 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:22.317611 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:22.318940 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:22.319487 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:22.319662 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:22.323162 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:22.323170 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:22.323181 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:22.323213 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:22.326054 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:22.326692 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:22.326698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:22.326700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:22.326840 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:22.327212 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:22.327216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:22.327220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:22.327685 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:22.327866 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:22.328386 (+ 520us) server_negotiation.cc:299] Negotiation successful
0619 04:36:22.328713 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":1491,"spinlock_wait_cycles":24320,"thread_start_us":1282,"threads_started":1}
I20260619 04:36:22.330185 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.311515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44377 (local address 127.0.0.1:40744)
0619 04:36:22.312384 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:22.312430 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:22.314391 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:22.315186 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:22.315193 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:22.315897 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:22.316232 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:22.316249 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:22.317747 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:22.317751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:22.318813 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:22.318825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:22.319710 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:22.319717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:22.319941 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:22.320659 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:22.320687 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:22.322960 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:22.326171 (+ 3211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:22.326181 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:22.326194 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:22.326579 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:22.326940 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:22.326945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:22.326949 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:22.327113 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:22.327810 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:22.327817 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:22.328029 (+ 212us) client_negotiation.cc:769] Sending connection context
0619 04:36:22.328249 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 04:36:22.328826 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":158,"thread_start_us":379,"threads_started":1}
I20260619 04:36:22.333340 13887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } has no permanent_uuid. Determining permanent_uuid...
I20260619 04:36:22.334249 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.333882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44327 (local address 127.0.0.1:59356)
0619 04:36:22.334030 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:22.334139 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 04:36:22.334484 13838 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44327: connect: Connection refused (error 111)
I20260619 04:36:22.335893 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.335568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44327 (local address 127.0.0.1:59370)
0619 04:36:22.335694 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:22.335768 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 04:36:22.336082 13887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44327: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44327: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260619 04:36:22.350749 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44327
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44377,127.0.97.61:44327,127.0.97.60:35879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:36:22.385383 13887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } attempt: 1
I20260619 04:36:22.386533 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.386115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44327 (local address 127.0.0.1:59376)
0619 04:36:22.386291 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:22.386383 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 04:36:22.387549 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.387193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44327 (local address 127.0.0.1:59390)
0619 04:36:22.387320 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:22.387385 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 04:36:22.387735 13887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44327: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44327: connect: Connection refused (error 111)
I20260619 04:36:22.455965 13887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } attempt: 2
I20260619 04:36:22.457109 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.456679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44327 (local address 127.0.0.1:59406)
0619 04:36:22.456857 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:22.456965 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 04:36:22.458106 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.457734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44327 (local address 127.0.0.1:59416)
0619 04:36:22.457885 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:22.457970 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 04:36:22.458276 13887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44327: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44327: connect: Connection refused (error 111)
I20260619 04:36:22.549499 13887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } attempt: 3
I20260619 04:36:22.550609 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.550186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44327 (local address 127.0.0.1:59420)
0619 04:36:22.550377 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:22.550474 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 04:36:22.551576 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.551219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44327 (local address 127.0.0.1:59426)
0619 04:36:22.551370 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:22.551435 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 04:36:22.551753 13887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44327: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44327: connect: Connection refused (error 111)
W20260619 04:36:22.663480 13894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:22.663800 13894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:22.663929 13894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:22.671160 13894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:36:22.671278 13894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:22.671355 13894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:22.671432 13894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:36:22.671489 13894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:36:22.681382 13894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44377,127.0.97.61:44327,127.0.97.60:35879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:22.682653 13894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:22.683898 13894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:22.694831 13902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:22.694893 13900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:22.694957 13887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } attempt: 4
W20260619 04:36:22.694876 13899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:22.695737 13894 server_base.cc:1034] running on GCE node
I20260619 04:36:22.696115 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.695595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44327 (local address 127.0.0.1:59440)
0619 04:36:22.695896 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:22.695987 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260619 04:36:22.696573 13894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:22.697114 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.696779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44327 (local address 127.0.0.1:59454)
0619 04:36:22.696911 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:22.696999 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 04:36:22.697273 13887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44327: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44327: connect: Connection refused (error 111)
I20260619 04:36:22.697978 13894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:22.699178 13894 hybrid_clock.cc:648] HybridClock initialized: now 1781843782699115 us; error 53 us; skew 500 ppm
I20260619 04:36:22.699536 13894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:22 dist-test-slave-n3tq krb5kdc[13807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:22.702462 13894 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260619 04:36:22.703801 13894 webserver.cc:460] Webserver started at http://127.0.97.61:46397/ using document root <none> and password file <none>
I20260619 04:36:22.704473 13894 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:22.704597 13894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:22.704866 13894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:36:22.707803 13894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-1/data/instance:
uuid: "fc228b4372c940d38b61d1e4658b9165"
format_stamp: "Formatted at 2026-06-19 04:36:22 on dist-test-slave-n3tq"
server_key: "66b24979c0f16451784a967627d5f792"
server_key_iv: "3ef83aef523979954f1210889af8c5e0"
server_key_version: "encryptionkey@0"
I20260619 04:36:22.708577 13894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-1/wal/instance:
uuid: "fc228b4372c940d38b61d1e4658b9165"
format_stamp: "Formatted at 2026-06-19 04:36:22 on dist-test-slave-n3tq"
server_key: "66b24979c0f16451784a967627d5f792"
server_key_iv: "3ef83aef523979954f1210889af8c5e0"
server_key_version: "encryptionkey@0"
I20260619 04:36:22.713515 13894 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 04:36:22.716760 13909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:22.718031 13894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:36:22.718187 13894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-1/wal
uuid: "fc228b4372c940d38b61d1e4658b9165"
format_stamp: "Formatted at 2026-06-19 04:36:22 on dist-test-slave-n3tq"
server_key: "66b24979c0f16451784a967627d5f792"
server_key_iv: "3ef83aef523979954f1210889af8c5e0"
server_key_version: "encryptionkey@0"
I20260619 04:36:22.718377 13894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:22.735498 13894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:22.739295 13894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:22.739539 13894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:22.756220 13894 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44327
I20260619 04:36:22.756222 13960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44327 every 8 connection(s)
I20260619 04:36:22.757603 13894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-1/data/info.pb
I20260619 04:36:22.759433 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 13894
I20260619 04:36:22.759606 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-1/wal/instance
I20260619 04:36:22.760147 388 external_mini_cluster.cc:1428] Setting key 4c986353eadb4e7b5260bc5c0dffddb8
I20260619 04:36:22.761786 13961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:36:22.769634 13961 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Jun 19 04:36:22 dist-test-slave-n3tq krb5kdc[13807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:36:22.794200 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.772203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40748 (local address 127.0.97.62:44377)
0619 04:36:22.772374 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:22.772379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:22.773755 (+ 1376us) server_negotiation.cc:407] Connection header received
0619 04:36:22.775586 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:22.775592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:22.775671 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:22.775756 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:22.777392 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:22.778086 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:22.779709 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:22.780142 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:22.780263 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:22.788654 (+ 8391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:22.788660 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:22.788663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:22.788699 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:22.791200 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:22.791940 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:22.791944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:22.791946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:22.792062 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:22.792490 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:22.792495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:22.792499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:22.792776 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:22.792927 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:22.793637 (+ 710us) server_negotiation.cc:299] Negotiation successful
0619 04:36:22.793879 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:36:22.795157 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.772785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44377 (local address 127.0.0.1:40748)
0619 04:36:22.773586 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:22.773624 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:22.775311 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:22.776000 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:22.776009 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:22.776644 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:22.777210 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:22.777227 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:22.778242 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:22.778246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:22.779568 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:22.779581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:22.780409 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:22.780418 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:22.780664 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:22.785758 (+ 5094us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:22.785791 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:22.788385 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:22.791381 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:22.791390 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:22.791405 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:22.791813 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:22.792164 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:22.792170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:22.792174 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:22.792360 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:22.792906 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:22.792914 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:22.793262 (+ 348us) client_negotiation.cc:769] Sending connection context
0619 04:36:22.793505 (+ 243us) client_negotiation.cc:240] Negotiation successful
0619 04:36:22.794027 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":355,"threads_started":1}
I20260619 04:36:22.797124 13961 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:36:22 dist-test-slave-n3tq krb5kdc[13807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260619 04:36:22.806726 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35879
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44377,127.0.97.61:44327,127.0.97.60:35879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:36:22.810333 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.797853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44327 (local address 127.0.0.1:59464)
0619 04:36:22.798007 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:22.798024 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:22.798121 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:22.798845 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:22.798850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:22.799246 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:22.799379 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:22.799387 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:22.800736 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:22.800740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:22.801626 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:22.801637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:22.802422 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:22.802455 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:22.802665 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:22.803179 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:22.803194 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:22.805083 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:22.808041 (+ 2958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:22.808046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:22.808050 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:22.808380 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:22.808719 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:22.808724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:22.808728 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:22.808865 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:22.809479 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:22.809484 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:22.809612 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 04:36:22.809724 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 04:36:22.809968 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 04:36:22.810586 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.797920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59464 (local address 127.0.97.61:44327)
0619 04:36:22.798467 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:22.798473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:22.798495 (+ 22us) server_negotiation.cc:407] Connection header received
0619 04:36:22.798533 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:22.798539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:22.798712 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:22.798848 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:22.799517 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:22.800599 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:22.801746 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:22.802277 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:22.802382 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:22.805272 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:22.805285 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:22.805301 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:22.805336 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:22.807936 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:22.808516 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:22.808525 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:22.808527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:22.808627 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:22.808958 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:22.808963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:22.808967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:22.809361 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:22.809532 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:22.809891 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 04:36:22.810096 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":274,"threads_started":1}
I20260619 04:36:22.812872 13961 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } has no permanent_uuid. Determining permanent_uuid...
I20260619 04:36:22.813961 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.813560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35879 (local address 127.0.0.1:35550)
0619 04:36:22.813711 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:22.813824 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 04:36:22.814218 13913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
I20260619 04:36:22.815631 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.815336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35879 (local address 127.0.0.1:35558)
0619 04:36:22.815461 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:22.815536 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 04:36:22.815804 13961 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35879: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
I20260619 04:36:22.866160 13961 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } attempt: 1
I20260619 04:36:22.867295 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.866826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35879 (local address 127.0.0.1:35562)
0619 04:36:22.867018 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:22.867129 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 04:36:22.868288 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.867942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35879 (local address 127.0.0.1:35572)
0619 04:36:22.868099 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:22.868165 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 04:36:22.868587 13961 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35879: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
I20260619 04:36:22.936785 13961 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } attempt: 2
I20260619 04:36:22.937814 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.937369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35879 (local address 127.0.0.1:35588)
0619 04:36:22.937560 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:22.937652 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 04:36:22.938766 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.938431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35879 (local address 127.0.0.1:35590)
0619 04:36:22.938569 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:22.938639 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 04:36:22.939033 13961 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35879: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
I20260619 04:36:22.996577 13887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } attempt: 5
Jun 19 04:36:22 dist-test-slave-n3tq krb5kdc[13807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 04:36:23.009673 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.997387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59480 (local address 127.0.97.61:44327)
0619 04:36:22.997569 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:22.997574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:22.997671 (+ 97us) server_negotiation.cc:407] Connection header received
0619 04:36:22.997859 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:22.997863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:22.997949 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:22.998033 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:22.998838 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:22.999608 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.000864 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.001382 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.001499 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:23.004547 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:23.004553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:23.004556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:23.004588 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:23.007192 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.007805 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.007809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.007812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.007942 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.008285 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.008289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.008291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.008622 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:23.008762 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:23.009114 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 04:36:23.009322 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 04:36:23.009667 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:22.997265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44327 (local address 127.0.0.1:59480)
0619 04:36:22.997566 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:22.997585 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:22.997727 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:22.998088 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:22.998093 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:22.998498 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:22.998678 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:22.998687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:22.999769 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:22.999775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:23.000702 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:23.000728 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.001544 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.001549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:23.001746 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:23.002335 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:23.002353 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:23.004339 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:23.007321 (+ 2982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:23.007326 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:23.007331 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:23.007690 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:23.008043 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:23.008048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:23.008052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:23.008182 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:23.008726 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:23.008731 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:23.008885 (+ 154us) client_negotiation.cc:769] Sending connection context
0619 04:36:23.009006 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 04:36:23.009297 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260619 04:36:23.010855 13887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } has no permanent_uuid. Determining permanent_uuid...
I20260619 04:36:23.011906 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.011531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35879 (local address 127.0.0.1:35606)
0619 04:36:23.011691 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:23.011775 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 04:36:23.012866 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.012533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35879 (local address 127.0.0.1:35620)
0619 04:36:23.012661 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:23.012756 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 04:36:23.013091 13887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35879: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
I20260619 04:36:23.030261 13961 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } attempt: 3
I20260619 04:36:23.031384 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.030911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35879 (local address 127.0.0.1:35630)
0619 04:36:23.031120 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:23.031219 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 04:36:23.032642 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.032170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35879 (local address 127.0.0.1:35634)
0619 04:36:23.032396 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:23.032509 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260619 04:36:23.032931 13961 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35879: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
I20260619 04:36:23.064313 13887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } attempt: 1
I20260619 04:36:23.065447 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.064997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35879 (local address 127.0.0.1:35650)
0619 04:36:23.065186 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:23.065272 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 04:36:23.066408 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.066088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35879 (local address 127.0.0.1:35660)
0619 04:36:23.066228 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:23.066304 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 04:36:23.066663 13887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35879: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
I20260619 04:36:23.134902 13887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } attempt: 2
I20260619 04:36:23.136039 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.135562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35879 (local address 127.0.0.1:35664)
0619 04:36:23.135781 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:23.135875 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 04:36:23.137043 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.136731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35879 (local address 127.0.0.1:35668)
0619 04:36:23.136856 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:23.136938 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 04:36:23.137243 13887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35879: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
I20260619 04:36:23.176144 13961 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } attempt: 4
I20260619 04:36:23.177335 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.176856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35879 (local address 127.0.0.1:35678)
0619 04:36:23.177081 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:23.177170 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 04:36:23.178285 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.177987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35879 (local address 127.0.0.1:35686)
0619 04:36:23.178114 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:23.178182 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 04:36:23.178486 13961 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35879: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
W20260619 04:36:23.219274 13967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:23.219563 13967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:23.219656 13967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:23.227185 13967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:36:23.227295 13967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:23.227340 13967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:23.227396 13967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:36:23.227439 13967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:36:23.237360 13967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44377,127.0.97.61:44327,127.0.97.60:35879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:23.238852 13967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:23.240209 13967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 04:36:23.243453 13887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } attempt: 3
I20260619 04:36:23.244572 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.244032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35879 (local address 127.0.0.1:35702)
0619 04:36:23.244293 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:23.244380 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 04:36:23.245704 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.245264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35879 (local address 127.0.0.1:35716)
0619 04:36:23.245486 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:23.245570 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260619 04:36:23.246013 13887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35879: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35879: connect: Connection refused (error 111)
W20260619 04:36:23.251998 13973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:23.252039 13976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:23.252065 13974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:23.252830 13967 server_base.cc:1034] running on GCE node
I20260619 04:36:23.253719 13967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:23.255142 13967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:23.256347 13967 hybrid_clock.cc:648] HybridClock initialized: now 1781843783256295 us; error 46 us; skew 500 ppm
I20260619 04:36:23.256776 13967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:23 dist-test-slave-n3tq krb5kdc[13807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:23.259933 13967 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260619 04:36:23.261466 13967 webserver.cc:460] Webserver started at http://127.0.97.60:43263/ using document root <none> and password file <none>
I20260619 04:36:23.262173 13967 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:23.262310 13967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:23.262603 13967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:36:23.265650 13967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-2/data/instance:
uuid: "7bcbf406bbe442de9d8bd99705956d24"
format_stamp: "Formatted at 2026-06-19 04:36:23 on dist-test-slave-n3tq"
server_key: "972dd0c45b0562bb39ef288f5939e44d"
server_key_iv: "a1cc55e660d396e96fe35390d974fae8"
server_key_version: "encryptionkey@0"
I20260619 04:36:23.266427 13967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-2/wal/instance:
uuid: "7bcbf406bbe442de9d8bd99705956d24"
format_stamp: "Formatted at 2026-06-19 04:36:23 on dist-test-slave-n3tq"
server_key: "972dd0c45b0562bb39ef288f5939e44d"
server_key_iv: "a1cc55e660d396e96fe35390d974fae8"
server_key_version: "encryptionkey@0"
I20260619 04:36:23.271437 13967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 04:36:23.274752 13983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:23.276178 13967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:36:23.276357 13967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-2/wal
uuid: "7bcbf406bbe442de9d8bd99705956d24"
format_stamp: "Formatted at 2026-06-19 04:36:23 on dist-test-slave-n3tq"
server_key: "972dd0c45b0562bb39ef288f5939e44d"
server_key_iv: "a1cc55e660d396e96fe35390d974fae8"
server_key_version: "encryptionkey@0"
I20260619 04:36:23.276574 13967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:23.308341 13967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:23.312400 13967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:23.312685 13967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:23.329555 13967 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:35879
I20260619 04:36:23.329586 14034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:35879 every 8 connection(s)
I20260619 04:36:23.330860 13967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-2/data/info.pb
I20260619 04:36:23.334538 14035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:36:23.338505 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 13967
I20260619 04:36:23.338702 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781843588881932-388-0/minicluster-data/master-2/wal/instance
I20260619 04:36:23.339236 388 external_mini_cluster.cc:1428] Setting key bd07faee712f489113c502a57313ce67
I20260619 04:36:23.342621 14035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:36:23 dist-test-slave-n3tq krb5kdc[13807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 04:36:23 dist-test-slave-n3tq krb5kdc[13807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:36:23.362267 14039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.346382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40758 (local address 127.0.97.62:44377)
0619 04:36:23.346922 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:23.346927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:23.347325 (+ 398us) server_negotiation.cc:407] Connection header received
0619 04:36:23.347507 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:23.347511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:23.347575 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:23.347663 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:23.349027 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.350031 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.351762 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.352189 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.352281 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:23.356027 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:23.356039 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:23.356045 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:23.356090 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:23.359366 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.360105 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.360109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.360112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.360254 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.360677 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.360681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.360682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.360991 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:23.361153 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:23.361790 (+ 637us) server_negotiation.cc:299] Negotiation successful
0619 04:36:23.361965 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":250,"threads_started":1}
I20260619 04:36:23.363348 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.346470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44377 (local address 127.0.0.1:40758)
0619 04:36:23.347193 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:23.347229 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:23.347386 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:23.347740 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:23.347748 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:23.348534 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:23.348859 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:23.348871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.350198 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.350201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:23.351619 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:23.351635 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.352425 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.352433 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:23.352649 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:23.353285 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:23.353326 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:23.355446 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:23.359503 (+ 4057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:23.359513 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:23.359520 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:23.359991 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:23.360369 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:23.360373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:23.360376 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:23.360569 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:23.361124 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:23.361132 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:23.361411 (+ 279us) client_negotiation.cc:769] Sending connection context
0619 04:36:23.361592 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 04:36:23.362017 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":131,"spinlock_wait_cycles":13056,"thread_start_us":376,"threads_started":1}
I20260619 04:36:23.363343 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.341189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35726 (local address 127.0.97.60:35879)
0619 04:36:23.341947 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:23.341959 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:23.342002 (+ 43us) server_negotiation.cc:407] Connection header received
0619 04:36:23.343563 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:23.343588 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:23.344011 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:23.344356 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:23.346139 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.347924 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.349573 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.350489 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.350634 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:23.355093 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:23.355101 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:23.355113 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:23.355144 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:23.358405 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.359316 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.359323 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.359331 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.359530 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.359951 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.359955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.359959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.360690 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:23.360917 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:23.361415 (+ 498us) server_negotiation.cc:299] Negotiation successful
0619 04:36:23.361891 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":559,"spinlock_wait_cycles":23936,"thread_start_us":364,"threads_started":1}
I20260619 04:36:23.365162 14035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:36:23 dist-test-slave-n3tq krb5kdc[13807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 19 04:36:23 dist-test-slave-n3tq krb5kdc[13807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:36:23.379001 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.366009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44327 (local address 127.0.0.1:59496)
0619 04:36:23.366165 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:23.366186 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:23.366281 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:23.366905 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:23.366909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:23.367307 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:23.367499 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:23.367511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.368559 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.368563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:23.369521 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:23.369536 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.370251 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.370257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:23.370500 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:23.371097 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:23.371114 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:23.373430 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:23.376766 (+ 3336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:23.376769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:23.376772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:23.377132 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:23.377523 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:23.377527 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:23.377529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:23.377668 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:23.378210 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:23.378214 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:23.378371 (+ 157us) client_negotiation.cc:769] Sending connection context
0619 04:36:23.378479 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 04:36:23.378686 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 04:36:23.379266 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.366124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.0.97.61:44327)
0619 04:36:23.366302 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:23.366308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:23.366327 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:36:23.366409 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:23.366413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:23.366483 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:23.366575 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:23.367678 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.368375 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.369676 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.370101 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.370198 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:23.373633 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:23.373639 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:23.373643 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:23.373676 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:23.376648 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.377242 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.377246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.377248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.377421 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.377766 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.377771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.377774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.378095 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:23.378247 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:23.378607 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 04:36:23.378826 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 04:36:23.380131 14035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } has no permanent_uuid. Determining permanent_uuid...
I20260619 04:36:23.381801 14039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.366580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40768 (local address 127.0.97.62:44377)
0619 04:36:23.366739 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:23.366743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:23.366757 (+ 14us) server_negotiation.cc:407] Connection header received
0619 04:36:23.366901 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:23.366905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:23.366979 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:23.367064 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:23.368534 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.369216 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.370878 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.371329 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.371427 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:23.375736 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:23.375742 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:23.375747 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:23.375786 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:23.378703 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.379403 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.379407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.379409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.379552 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.379957 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.379963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.379967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.380329 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:23.380592 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:23.381277 (+ 685us) server_negotiation.cc:299] Negotiation successful
0619 04:36:23.381489 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 04:36:23 dist-test-slave-n3tq krb5kdc[13807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 04:36:23 dist-test-slave-n3tq krb5kdc[13807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 04:36:23.392431 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.380822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35879 (local address 127.0.0.1:35732)
0619 04:36:23.381001 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:23.381020 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:23.381114 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:23.381445 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:23.381450 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:23.381867 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:23.382045 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:23.382056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.383113 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.383116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:23.384063 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:23.384074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.384763 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.384768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:23.384958 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:23.385583 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:23.385600 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:23.387582 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:23.390237 (+ 2655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:23.390242 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:23.390246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:23.390565 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:23.390863 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:23.390868 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:23.390872 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:23.390992 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:23.391557 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:23.391561 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:23.391694 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 04:36:23.391809 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 04:36:23.392047 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 04:36:23.392439 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.380993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35732 (local address 127.0.97.60:35879)
0619 04:36:23.381204 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:23.381209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:23.381235 (+ 26us) server_negotiation.cc:407] Connection header received
0619 04:36:23.381271 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:23.381274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:23.381323 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:23.381393 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:23.382184 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.382993 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.384178 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.384637 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.384734 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:23.387751 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:23.387757 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:23.387760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:23.387786 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:23.390127 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.390663 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.390668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.390672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.390774 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.391081 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.391085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.391089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.391421 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:23.391574 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:23.391924 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 04:36:23.392184 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":111}
I20260619 04:36:23.397195 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.383666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59500 (local address 127.0.97.61:44327)
0619 04:36:23.383840 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:23.383846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:23.383860 (+ 14us) server_negotiation.cc:407] Connection header received
0619 04:36:23.383897 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:23.383900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:23.383949 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:23.384033 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:23.385243 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.385928 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.387414 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.387822 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.387928 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:23.392116 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:23.392124 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:23.392129 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:23.392167 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:23.394783 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.395541 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.395545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.395547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.395655 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.396007 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.396010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.396012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.396261 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:23.396524 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:23.396795 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 04:36:23.396953 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 04:36:23.397886 14035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24: Bootstrap starting.
I20260619 04:36:23.401262 14035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24: Neither blocks nor log segments found. Creating new log.
I20260619 04:36:23.401666 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:36:23.402311 14035 log.cc:824] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24: Log is configured to *not* fsync() on all Append() calls
I20260619 04:36:23.405088 14035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24: No bootstrap required, opened a new log
I20260619 04:36:23.416208 14035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2299da07d0814549b99bf5a6a41bd6b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 } } peers { permanent_uuid: "fc228b4372c940d38b61d1e4658b9165" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } } peers { permanent_uuid: "7bcbf406bbe442de9d8bd99705956d24" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } }
I20260619 04:36:23.418202 14035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:36:23.418375 14035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bcbf406bbe442de9d8bd99705956d24, State: Initialized, Role: FOLLOWER
I20260619 04:36:23.419096 14035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2299da07d0814549b99bf5a6a41bd6b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 } } peers { permanent_uuid: "fc228b4372c940d38b61d1e4658b9165" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } } peers { permanent_uuid: "7bcbf406bbe442de9d8bd99705956d24" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } }
I20260619 04:36:23.423210 13887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } attempt: 4
I20260619 04:36:23.426337 14049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2299da07d0814549b99bf5a6a41bd6b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 } } peers { permanent_uuid: "fc228b4372c940d38b61d1e4658b9165" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } } peers { permanent_uuid: "7bcbf406bbe442de9d8bd99705956d24" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } } }
I20260619 04:36:23.426759 14049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:36:23.428664 14035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:36:23.435904 14039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.406852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40784 (local address 127.0.97.62:44377)
0619 04:36:23.407046 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:23.407053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:23.417006 (+ 9953us) server_negotiation.cc:407] Connection header received
0619 04:36:23.417176 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:23.417183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:23.417266 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:23.417359 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:23.420162 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.421400 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.423739 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.424488 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.424589 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:23.428592 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:23.428599 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:23.428603 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:23.428653 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:23.432087 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.432957 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.432963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.432966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.433119 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.433606 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.433611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.433615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.433958 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:23.434192 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:23.435285 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0619 04:36:23.435515 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 04:36:23.436049 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.406500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35740 (local address 127.0.97.60:35879)
0619 04:36:23.406654 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:23.406660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:23.419191 (+ 12531us) server_negotiation.cc:407] Connection header received
0619 04:36:23.419328 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:23.419333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:23.419406 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:23.419508 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:23.421560 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.422630 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.424870 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.425633 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.425729 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:23.429407 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:23.429414 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:23.429422 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:23.429458 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:23.432760 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.433699 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.433705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.433709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.433887 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.434372 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.434378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.434382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.434763 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:23.434979 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:23.435516 (+ 537us) server_negotiation.cc:299] Negotiation successful
0619 04:36:23.435712 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 04:36:23 dist-test-slave-n3tq krb5kdc[13807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 04:36:23.443565 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.406939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.0.97.61:44327)
0619 04:36:23.412033 (+ 5094us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:23.412039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:23.417674 (+ 5635us) server_negotiation.cc:407] Connection header received
0619 04:36:23.417822 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:23.417828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:23.417913 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:23.418016 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:23.421121 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.422325 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.431656 (+ 9331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.432304 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.432406 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:23.435660 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:23.435667 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:23.435673 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:23.435711 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:23.440273 (+ 4562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.441135 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.441142 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.441145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.441293 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.441735 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.441740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.441743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.442099 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:23.442324 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:23.442870 (+ 546us) server_negotiation.cc:299] Negotiation successful
0619 04:36:23.443197 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4972}
I20260619 04:36:23.451913 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.424436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35744 (local address 127.0.97.60:35879)
0619 04:36:23.430668 (+ 6232us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:23.430674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:23.430697 (+ 23us) server_negotiation.cc:407] Connection header received
0619 04:36:23.430748 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:23.430753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:23.430823 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:23.431910 (+ 1087us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:23.432202 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.433366 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.436480 (+ 3114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.437099 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.437213 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:23.442141 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:23.442159 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:23.442166 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:23.442239 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:23.447610 (+ 5371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.449227 (+ 1617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.449237 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.449242 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.449477 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.450184 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.450190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.450193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.450624 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:23.450772 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:23.451178 (+ 406us) server_negotiation.cc:299] Negotiation successful
0619 04:36:23.451433 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5982,"spinlock_wait_cycles":207104,"thread_start_us":5792,"threads_started":1}
I20260619 04:36:23.452698 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.423774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35879 (local address 127.0.0.1:35744)
0619 04:36:23.424132 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:23.424151 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:23.424250 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:23.431016 (+ 6766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:23.431021 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:23.431505 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:23.431761 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:23.431774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.434665 (+ 2891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.434671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:23.436074 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:23.436089 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.437228 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.437234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:23.437418 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:23.438276 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:23.438327 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:23.441670 (+ 3343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:23.447985 (+ 6315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:23.447995 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:23.448001 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:23.448938 (+ 937us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:23.449749 (+ 811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:23.449754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:23.449757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:23.449976 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:23.450777 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:23.450782 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:23.450948 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 04:36:23.451072 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 04:36:23.452303 (+ 1231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233}
I20260619 04:36:23.458649 13887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6: Bootstrap starting.
W20260619 04:36:23.461365 14061 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 04:36:23.461611 14061 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 04:36:23.462635 13887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6: Neither blocks nor log segments found. Creating new log.
I20260619 04:36:23.464040 13887 log.cc:824] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6: Log is configured to *not* fsync() on all Append() calls
I20260619 04:36:23.467252 13887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6: No bootstrap required, opened a new log
I20260619 04:36:23.458669 13967 master_runner.cc:186] Error getting master registration for 127.0.97.62:44377: OK, instance_id { permanent_uuid: "2299da07d0814549b99bf5a6a41bd6b6" instance_seqno: 1781843782290709 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 04:36:23.467747 13967 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 04:36:23.477694 13961 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } attempt: 5
I20260619 04:36:23.477838 13887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2299da07d0814549b99bf5a6a41bd6b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 } } peers { permanent_uuid: "fc228b4372c940d38b61d1e4658b9165" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } } peers { permanent_uuid: "7bcbf406bbe442de9d8bd99705956d24" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } }
I20260619 04:36:23.478231 13887 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:36:23.478360 13887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2299da07d0814549b99bf5a6a41bd6b6, State: Initialized, Role: FOLLOWER
I20260619 04:36:23.479719 13887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2299da07d0814549b99bf5a6a41bd6b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 } } peers { permanent_uuid: "fc228b4372c940d38b61d1e4658b9165" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } } peers { permanent_uuid: "7bcbf406bbe442de9d8bd99705956d24" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } }
I20260619 04:36:23.480885 14064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2299da07d0814549b99bf5a6a41bd6b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 } } peers { permanent_uuid: "fc228b4372c940d38b61d1e4658b9165" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } } peers { permanent_uuid: "7bcbf406bbe442de9d8bd99705956d24" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } } }
I20260619 04:36:23.481410 14064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:36:23.482219 13887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6 [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 04:36:23 dist-test-slave-n3tq krb5kdc[13807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260619 04:36:23.494089 14075 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 04:36:23.494263 14075 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 04:36:23.495673 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.478273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35879 (local address 127.0.0.1:35746)
0619 04:36:23.478650 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:23.478671 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:23.478775 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:23.479626 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:23.479630 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:23.480150 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:23.480378 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:23.480392 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.481867 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.481872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:23.482798 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:23.482809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.483656 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.483663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:23.483852 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:23.484517 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:23.484540 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:23.487102 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:23.492957 (+ 5855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:23.492962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:23.492967 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:23.493465 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:23.493878 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:23.493883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:23.493887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:23.494042 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:23.494710 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:23.494716 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:23.494869 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 04:36:23.495012 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 04:36:23.495291 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248}
I20260619 04:36:23.496534 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:23.479091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35746 (local address 127.0.97.60:35879)
0619 04:36:23.479282 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:23.479288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:23.479306 (+ 18us) server_negotiation.cc:407] Connection header received
0619 04:36:23.479366 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:23.479371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:23.479443 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:23.479547 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:23.480689 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.481743 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.482930 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:23.483339 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:23.483453 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:23.487359 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:23.487365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:23.487370 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:23.487407 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:23.492807 (+ 5400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.493603 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.493609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.493612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.493757 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:23.494159 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:23.494164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:23.494168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:23.494523 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:23.494653 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:23.495952 (+ 1299us) server_negotiation.cc:299] Negotiation successful
0619 04:36:23.496174 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 04:36:23.491187 13820 master_runner.cc:186] Error getting master registration for 127.0.97.61:44327: OK, instance_id { permanent_uuid: "fc228b4372c940d38b61d1e4658b9165" instance_seqno: 1781843782751500 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 04:36:23.501739 13820 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 04:36:23.502223 13961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165: Bootstrap starting.
I20260619 04:36:23.506137 13961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165: Neither blocks nor log segments found. Creating new log.
I20260619 04:36:23.507117 13961 log.cc:824] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165: Log is configured to *not* fsync() on all Append() calls
I20260619 04:36:23.509618 13961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165: No bootstrap required, opened a new log
I20260619 04:36:23.517644 13961 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2299da07d0814549b99bf5a6a41bd6b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 } } peers { permanent_uuid: "fc228b4372c940d38b61d1e4658b9165" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } } peers { permanent_uuid: "7bcbf406bbe442de9d8bd99705956d24" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } }
I20260619 04:36:23.517949 13961 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:36:23.518061 13961 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc228b4372c940d38b61d1e4658b9165, State: Initialized, Role: FOLLOWER
I20260619 04:36:23.518611 13961 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2299da07d0814549b99bf5a6a41bd6b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 } } peers { permanent_uuid: "fc228b4372c940d38b61d1e4658b9165" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } } peers { permanent_uuid: "7bcbf406bbe442de9d8bd99705956d24" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } }
I20260619 04:36:23.519465 14078 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2299da07d0814549b99bf5a6a41bd6b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 } } peers { permanent_uuid: "fc228b4372c940d38b61d1e4658b9165" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } } peers { permanent_uuid: "7bcbf406bbe442de9d8bd99705956d24" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } } }
I20260619 04:36:23.519907 14078 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:36:23.520721 13961 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 04:36:23.531639 13841 tablet.cc:2369] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:36:23.539984 13894 master_runner.cc:263] No leader master found from master fc228b4372c940d38b61d1e4658b9165
I20260619 04:36:23.540146 13894 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 04:36:23.546921 14089 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 04:36:23.547156 14089 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 04:36:23.564718 13989 tablet.cc:2369] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:36:23.742180 13915 tablet.cc:2369] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:36:23.820770 14078 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:36:23.821171 14078 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2299da07d0814549b99bf5a6a41bd6b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 } } peers { permanent_uuid: "fc228b4372c940d38b61d1e4658b9165" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } } peers { permanent_uuid: "7bcbf406bbe442de9d8bd99705956d24" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } }
I20260619 04:36:23.823000 14078 leader_election.cc:288] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2299da07d0814549b99bf5a6a41bd6b6 (127.0.97.62:44377), 7bcbf406bbe442de9d8bd99705956d24 (127.0.97.60:35879)
I20260619 04:36:23.824349 13862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc228b4372c940d38b61d1e4658b9165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2299da07d0814549b99bf5a6a41bd6b6" is_pre_election: true
I20260619 04:36:23.824978 13862 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc228b4372c940d38b61d1e4658b9165 in term 0.
I20260619 04:36:23.826104 13911 leader_election.cc:302] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2299da07d0814549b99bf5a6a41bd6b6, fc228b4372c940d38b61d1e4658b9165; no voters:
I20260619 04:36:23.827212 14010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc228b4372c940d38b61d1e4658b9165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bcbf406bbe442de9d8bd99705956d24" is_pre_election: true
I20260619 04:36:23.827847 14010 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc228b4372c940d38b61d1e4658b9165 in term 0.
I20260619 04:36:23.833293 14078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:36:23.833470 14078 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:36:23.833576 14078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:36:23.837015 14078 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2299da07d0814549b99bf5a6a41bd6b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 } } peers { permanent_uuid: "fc228b4372c940d38b61d1e4658b9165" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } } peers { permanent_uuid: "7bcbf406bbe442de9d8bd99705956d24" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } }
I20260619 04:36:23.838166 13862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc228b4372c940d38b61d1e4658b9165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2299da07d0814549b99bf5a6a41bd6b6"
I20260619 04:36:23.838387 13862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:36:23.841907 13862 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc228b4372c940d38b61d1e4658b9165 in term 1.
I20260619 04:36:23.842415 13911 leader_election.cc:302] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2299da07d0814549b99bf5a6a41bd6b6, fc228b4372c940d38b61d1e4658b9165; no voters:
I20260619 04:36:23.846020 14010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc228b4372c940d38b61d1e4658b9165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bcbf406bbe442de9d8bd99705956d24"
I20260619 04:36:23.846262 14010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:36:23.850049 14010 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc228b4372c940d38b61d1e4658b9165 in term 1.
I20260619 04:36:23.850841 14078 leader_election.cc:288] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [CANDIDATE]: Term 1 election: Requested vote from peers 2299da07d0814549b99bf5a6a41bd6b6 (127.0.97.62:44377), 7bcbf406bbe442de9d8bd99705956d24 (127.0.97.60:35879)
I20260619 04:36:23.851251 14078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:36:23.851697 14078 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [term 1 LEADER]: Becoming Leader. State: Replica: fc228b4372c940d38b61d1e4658b9165, State: Running, Role: LEADER
I20260619 04:36:23.852631 14078 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2299da07d0814549b99bf5a6a41bd6b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 } } peers { permanent_uuid: "fc228b4372c940d38b61d1e4658b9165" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } } peers { permanent_uuid: "7bcbf406bbe442de9d8bd99705956d24" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } }
I20260619 04:36:23.871784 14091 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc228b4372c940d38b61d1e4658b9165. Latest consensus state: current_term: 1 leader_uuid: "fc228b4372c940d38b61d1e4658b9165" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2299da07d0814549b99bf5a6a41bd6b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 } } peers { permanent_uuid: "fc228b4372c940d38b61d1e4658b9165" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } } peers { permanent_uuid: "7bcbf406bbe442de9d8bd99705956d24" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } } }
I20260619 04:36:23.872377 14091 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [sys.catalog]: This master's current role is: LEADER
I20260619 04:36:23.879570 14093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:36:23.882671 14093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:36:23.898643 14010 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24 [term 1 FOLLOWER]: Refusing update from remote peer fc228b4372c940d38b61d1e4658b9165: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:36:23.903180 14091 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bcbf406bbe442de9d8bd99705956d24" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:36:23.911209 13862 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6 [term 1 FOLLOWER]: Refusing update from remote peer fc228b4372c940d38b61d1e4658b9165: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:36:23.912139 14091 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2299da07d0814549b99bf5a6a41bd6b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:36:23.946238 14049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc228b4372c940d38b61d1e4658b9165. Latest consensus state: current_term: 1 leader_uuid: "fc228b4372c940d38b61d1e4658b9165" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2299da07d0814549b99bf5a6a41bd6b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 } } peers { permanent_uuid: "fc228b4372c940d38b61d1e4658b9165" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } } peers { permanent_uuid: "7bcbf406bbe442de9d8bd99705956d24" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } } }
I20260619 04:36:23.946662 14049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:36:23.949185 14064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc228b4372c940d38b61d1e4658b9165. Latest consensus state: current_term: 1 leader_uuid: "fc228b4372c940d38b61d1e4658b9165" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2299da07d0814549b99bf5a6a41bd6b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 } } peers { permanent_uuid: "fc228b4372c940d38b61d1e4658b9165" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } } peers { permanent_uuid: "7bcbf406bbe442de9d8bd99705956d24" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } } }
I20260619 04:36:23.949525 14064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:36:23.956437 14064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc228b4372c940d38b61d1e4658b9165" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2299da07d0814549b99bf5a6a41bd6b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 } } peers { permanent_uuid: "fc228b4372c940d38b61d1e4658b9165" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } } peers { permanent_uuid: "7bcbf406bbe442de9d8bd99705956d24" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } } }
I20260619 04:36:23.956755 14064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:36:23.959455 14091 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc228b4372c940d38b61d1e4658b9165" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2299da07d0814549b99bf5a6a41bd6b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 } } peers { permanent_uuid: "fc228b4372c940d38b61d1e4658b9165" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } } peers { permanent_uuid: "7bcbf406bbe442de9d8bd99705956d24" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } } }
I20260619 04:36:23.959733 14091 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [sys.catalog]: This master's current role is: LEADER
I20260619 04:36:23.962800 14095 mvcc.cc:205] Tried to move back new op lower bound from 7298432138836664320 to 7298432138673999872. Current Snapshot: MvccSnapshot[applied={T|T < 7298432138836664320}]
I20260619 04:36:23.969681 14049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc228b4372c940d38b61d1e4658b9165" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2299da07d0814549b99bf5a6a41bd6b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 } } peers { permanent_uuid: "fc228b4372c940d38b61d1e4658b9165" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } } peers { permanent_uuid: "7bcbf406bbe442de9d8bd99705956d24" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } } }
I20260619 04:36:23.969942 14049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:36:23.970847 14093 catalog_manager.cc:1347] Generated new cluster ID: c0e9bc6cf6fb4e00bc24140aef83bff1
I20260619 04:36:23.970961 14093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:36:23.998035 14078 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc228b4372c940d38b61d1e4658b9165" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2299da07d0814549b99bf5a6a41bd6b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44377 } } peers { permanent_uuid: "fc228b4372c940d38b61d1e4658b9165" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44327 } } peers { permanent_uuid: "7bcbf406bbe442de9d8bd99705956d24" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35879 } } }
I20260619 04:36:23.998526 14078 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165 [sys.catalog]: This master's current role is: LEADER
I20260619 04:36:23.998860 14093 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:36:24.000061 14093 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:36:24.014392 14093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc228b4372c940d38b61d1e4658b9165: Generated new TSK 0
I20260619 04:36:24.015450 14093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:36:24.463594 14061 catalog_manager.cc:1259] Loaded cluster ID: c0e9bc6cf6fb4e00bc24140aef83bff1
I20260619 04:36:24.463699 14061 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24: loading cluster ID for follower catalog manager: success
I20260619 04:36:24.466202 14061 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24: acquiring CA information for follower catalog manager: success
I20260619 04:36:24.467978 14061 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7bcbf406bbe442de9d8bd99705956d24: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 04:36:24.496152 14075 catalog_manager.cc:1259] Loaded cluster ID: c0e9bc6cf6fb4e00bc24140aef83bff1
I20260619 04:36:24.496248 14075 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6: loading cluster ID for follower catalog manager: success
I20260619 04:36:24.498713 14075 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6: acquiring CA information for follower catalog manager: success
I20260619 04:36:24.500422 14075 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2299da07d0814549b99bf5a6a41bd6b6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 04:36:24.596938 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 13820
I20260619 04:36:24.598163 13882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:24.706885 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 13820
I20260619 04:36:24.724109 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 13894
I20260619 04:36:24.725158 13956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:24.831212 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 13894
I20260619 04:36:24.847749 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 13967
I20260619 04:36:24.848817 14030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:24.953735 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 13967
2026-06-19T04:36:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4267 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:36:25 dist-test-slave-n3tq krb5kdc[14112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:36:25 dist-test-slave-n3tq krb5kdc[14112](info): set up 2 sockets
Jun 19 04:36:25 dist-test-slave-n3tq krb5kdc[14112](info): commencing operation
krb5kdc: starting...
W20260619 04:36:25.956552 388 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:36:26 dist-test-slave-n3tq krb5kdc[14112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:36:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:36:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:36:26.181674 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43505
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:36:26.497229 14125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:26.497522 14125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:26.497609 14125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:26.504683 14125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:36:26.504786 14125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:26.504838 14125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:26.504895 14125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:36:26.504947 14125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:36:26.514611 14125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:26.515877 14125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:26.517171 14125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:26.527807 14130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:26.527894 14133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:26.527853 14131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:26.528765 14125 server_base.cc:1034] running on GCE node
I20260619 04:36:26.529487 14125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:26.530805 14125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:26.532007 14125 hybrid_clock.cc:648] HybridClock initialized: now 1781843786531960 us; error 31 us; skew 500 ppm
I20260619 04:36:26.532326 14125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:26 dist-test-slave-n3tq krb5kdc[14112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:26.535173 14125 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:36:26.536530 14125 webserver.cc:460] Webserver started at http://127.0.97.62:35365/ using document root <none> and password file <none>
I20260619 04:36:26.537176 14125 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:26.537284 14125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:26.537562 14125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:36:26.540305 14125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "8ec5bab8605144a7b0fc685e02550b9c"
format_stamp: "Formatted at 2026-06-19 04:36:26 on dist-test-slave-n3tq"
server_key: "acf2f03acacbb6215c156ea3e9cc13bf"
server_key_iv: "ce8834ae5bdc4809adddd2152bb5e7f6"
server_key_version: "encryptionkey@0"
I20260619 04:36:26.541046 14125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "8ec5bab8605144a7b0fc685e02550b9c"
format_stamp: "Formatted at 2026-06-19 04:36:26 on dist-test-slave-n3tq"
server_key: "acf2f03acacbb6215c156ea3e9cc13bf"
server_key_iv: "ce8834ae5bdc4809adddd2152bb5e7f6"
server_key_version: "encryptionkey@0"
I20260619 04:36:26.545770 14125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 04:36:26.548951 14140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:26.550190 14125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 04:36:26.550334 14125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "8ec5bab8605144a7b0fc685e02550b9c"
format_stamp: "Formatted at 2026-06-19 04:36:26 on dist-test-slave-n3tq"
server_key: "acf2f03acacbb6215c156ea3e9cc13bf"
server_key_iv: "ce8834ae5bdc4809adddd2152bb5e7f6"
server_key_version: "encryptionkey@0"
I20260619 04:36:26.550503 14125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:26.567589 14125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:26.571306 14125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:26.571563 14125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:26.588235 14125 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43505
I20260619 04:36:26.588265 14191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43505 every 8 connection(s)
I20260619 04:36:26.589568 14125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:36:26.592499 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 14125
I20260619 04:36:26.592707 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:36:26.593590 388 external_mini_cluster.cc:1428] Setting key 86d8da10e0e19c0b763f4489c3e63995
I20260619 04:36:26.593691 14192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:36:26.607509 14192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c: Bootstrap starting.
Jun 19 04:36:26 dist-test-slave-n3tq krb5kdc[14112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:36:26.611496 14192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c: Neither blocks nor log segments found. Creating new log.
I20260619 04:36:26.612823 14192 log.cc:824] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c: Log is configured to *not* fsync() on all Append() calls
I20260619 04:36:26.616003 14192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c: No bootstrap required, opened a new log
I20260619 04:36:26.617636 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:26.595289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38986 (local address 127.0.97.62:43505)
0619 04:36:26.596027 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:26.596039 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:26.596082 (+ 43us) server_negotiation.cc:407] Connection header received
0619 04:36:26.598086 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:26.598106 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:26.598485 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:26.598815 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:26.600555 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:26.601783 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:26.603567 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:26.604306 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:26.604507 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:26.609402 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:26.609413 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:26.609431 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:26.609478 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:26.613738 (+ 4260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:26.614493 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:26.614500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:26.614506 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:26.614691 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:26.615090 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:26.615093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:26.615095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:26.615610 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:26.615818 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:26.616301 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 04:36:26.616681 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":347,"threads_started":1}
I20260619 04:36:26.621253 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:36:26.623951 14192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec5bab8605144a7b0fc685e02550b9c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43505 } }
I20260619 04:36:26.624327 14192 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:36:26.624416 14192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ec5bab8605144a7b0fc685e02550b9c, State: Initialized, Role: FOLLOWER
I20260619 04:36:26.624946 14192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec5bab8605144a7b0fc685e02550b9c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43505 } }
I20260619 04:36:26.625164 14192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:36:26.625284 14192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:36:26.625440 14192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:36:26.627496 14192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec5bab8605144a7b0fc685e02550b9c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43505 } }
I20260619 04:36:26.627974 14192 leader_election.cc:302] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ec5bab8605144a7b0fc685e02550b9c; no voters:
I20260619 04:36:26.628556 14192 leader_election.cc:288] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:36:26.628645 14197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:36:26.629233 14197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c [term 1 LEADER]: Becoming Leader. State: Replica: 8ec5bab8605144a7b0fc685e02550b9c, State: Running, Role: LEADER
I20260619 04:36:26.629782 14197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec5bab8605144a7b0fc685e02550b9c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43505 } }
I20260619 04:36:26.630501 14192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:36:26.632375 14198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ec5bab8605144a7b0fc685e02550b9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec5bab8605144a7b0fc685e02550b9c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43505 } } }
I20260619 04:36:26.632782 14198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c [sys.catalog]: This master's current role is: LEADER
I20260619 04:36:26.633244 14199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ec5bab8605144a7b0fc685e02550b9c. Latest consensus state: current_term: 1 leader_uuid: "8ec5bab8605144a7b0fc685e02550b9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec5bab8605144a7b0fc685e02550b9c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43505 } } }
I20260619 04:36:26.633576 14199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c [sys.catalog]: This master's current role is: LEADER
I20260619 04:36:26.641162 14206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:36:26.645319 14206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:36:26.653024 14206 catalog_manager.cc:1347] Generated new cluster ID: a04f2751a07d42b784c2313dadc16e9b
I20260619 04:36:26.653143 14206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:36:26.665472 14206 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:36:26.666759 14206 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:36:26.672883 14206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ec5bab8605144a7b0fc685e02550b9c: Generated new TSK 0
I20260619 04:36:26.673882 14206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:36:26.731948 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 14125
I20260619 04:36:26.733120 14187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:26.829177 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 14125
2026-06-19T04:36:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1871 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:36:26 dist-test-slave-n3tq krb5kdc[14220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:36:26 dist-test-slave-n3tq krb5kdc[14220](info): set up 2 sockets
Jun 19 04:36:26 dist-test-slave-n3tq krb5kdc[14220](info): commencing operation
krb5kdc: starting...
W20260619 04:36:27.832621 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:36:27 dist-test-slave-n3tq krb5kdc[14220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:36:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:36:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:36:28.037498 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41103
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:36:28.354511 14233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:28.354799 14233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:28.354923 14233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:28.362114 14233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:36:28.362221 14233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:28.362295 14233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:28.362363 14233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:36:28.362419 14233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:36:28.372174 14233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:28.373490 14233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:28.374805 14233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:28.385944 14239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:28.385941 14238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:28.386096 14241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:28.386818 14233 server_base.cc:1034] running on GCE node
I20260619 04:36:28.387609 14233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:28.389103 14233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:28.390316 14233 hybrid_clock.cc:648] HybridClock initialized: now 1781843788390263 us; error 50 us; skew 500 ppm
I20260619 04:36:28.390650 14233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:28 dist-test-slave-n3tq krb5kdc[14220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:28.396445 14233 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:36:28.398186 14233 webserver.cc:460] Webserver started at http://127.0.97.62:36555/ using document root <none> and password file <none>
I20260619 04:36:28.398908 14233 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:28.399029 14233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:28.399329 14233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:36:28.402388 14233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "d6c6cbf4cf7d4996840f70e43e689043"
format_stamp: "Formatted at 2026-06-19 04:36:28 on dist-test-slave-n3tq"
server_key: "492fe3b4eb8d5b9866b9e0d7c5ab9deb"
server_key_iv: "3c83bf2d423230d62bb83c209efacc9f"
server_key_version: "encryptionkey@0"
I20260619 04:36:28.403144 14233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "d6c6cbf4cf7d4996840f70e43e689043"
format_stamp: "Formatted at 2026-06-19 04:36:28 on dist-test-slave-n3tq"
server_key: "492fe3b4eb8d5b9866b9e0d7c5ab9deb"
server_key_iv: "3c83bf2d423230d62bb83c209efacc9f"
server_key_version: "encryptionkey@0"
I20260619 04:36:28.408432 14233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 04:36:28.411943 14248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:28.413335 14233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 04:36:28.413486 14233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "d6c6cbf4cf7d4996840f70e43e689043"
format_stamp: "Formatted at 2026-06-19 04:36:28 on dist-test-slave-n3tq"
server_key: "492fe3b4eb8d5b9866b9e0d7c5ab9deb"
server_key_iv: "3c83bf2d423230d62bb83c209efacc9f"
server_key_version: "encryptionkey@0"
I20260619 04:36:28.413668 14233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:28.428145 14233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:28.432230 14233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:28.432518 14233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:28.449688 14233 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41103
I20260619 04:36:28.449707 14299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41103 every 8 connection(s)
I20260619 04:36:28.451048 14233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:36:28.454694 14300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:36:28.457489 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 14233
I20260619 04:36:28.457690 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:36:28.458479 388 external_mini_cluster.cc:1428] Setting key 6305c99ec1a771b24c93cafdef81b7c1
I20260619 04:36:28.467180 14300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043: Bootstrap starting.
I20260619 04:36:28.470850 14300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043: Neither blocks nor log segments found. Creating new log.
I20260619 04:36:28.472138 14300 log.cc:824] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043: Log is configured to *not* fsync() on all Append() calls
Jun 19 04:36:28 dist-test-slave-n3tq krb5kdc[14220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:36:28.475427 14300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043: No bootstrap required, opened a new log
I20260619 04:36:28.482077 14303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:28.460773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54108 (local address 127.0.97.62:41103)
0619 04:36:28.461583 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:28.461596 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:28.461644 (+ 48us) server_negotiation.cc:407] Connection header received
0619 04:36:28.463291 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:28.463313 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:28.463729 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:28.464052 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:28.465687 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:28.467022 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:28.468783 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:28.469499 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:28.469632 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:28.474613 (+ 4981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:28.474621 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:28.474636 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:28.474675 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:28.477712 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:28.478526 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:28.478533 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:28.478540 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:28.478708 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:28.479148 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:28.479152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:28.479154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:28.479725 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:28.479970 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:28.480551 (+ 581us) server_negotiation.cc:299] Negotiation successful
0619 04:36:28.481006 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":365,"threads_started":1}
I20260619 04:36:28.484275 14300 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c6cbf4cf7d4996840f70e43e689043" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41103 } }
I20260619 04:36:28.484684 14300 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:36:28.484782 14300 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c6cbf4cf7d4996840f70e43e689043, State: Initialized, Role: FOLLOWER
I20260619 04:36:28.485404 14300 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c6cbf4cf7d4996840f70e43e689043" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41103 } }
I20260619 04:36:28.485616 14300 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:36:28.485737 14300 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:36:28.485874 14300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:36:28.487939 14300 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c6cbf4cf7d4996840f70e43e689043" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41103 } }
I20260619 04:36:28.488423 14300 leader_election.cc:302] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6c6cbf4cf7d4996840f70e43e689043; no voters:
I20260619 04:36:28.489094 14300 leader_election.cc:288] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:36:28.489192 14305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:36:28.489872 14305 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 1 LEADER]: Becoming Leader. State: Replica: d6c6cbf4cf7d4996840f70e43e689043, State: Running, Role: LEADER
I20260619 04:36:28.490371 14305 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c6cbf4cf7d4996840f70e43e689043" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41103 } }
I20260619 04:36:28.491148 14300 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:36:28.493187 14306 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6c6cbf4cf7d4996840f70e43e689043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c6cbf4cf7d4996840f70e43e689043" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41103 } } }
I20260619 04:36:28.493577 14306 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [sys.catalog]: This master's current role is: LEADER
I20260619 04:36:28.493642 14307 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6c6cbf4cf7d4996840f70e43e689043. Latest consensus state: current_term: 1 leader_uuid: "d6c6cbf4cf7d4996840f70e43e689043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c6cbf4cf7d4996840f70e43e689043" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41103 } } }
I20260619 04:36:28.493970 14307 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [sys.catalog]: This master's current role is: LEADER
I20260619 04:36:28.495983 14314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:36:28.501833 14314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:36:28.510886 14314 catalog_manager.cc:1347] Generated new cluster ID: 8f56ce11127643ef9b1d883a073e62f6
I20260619 04:36:28.510998 14314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:36:28.523190 14314 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:36:28.525063 14314 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:36:28.537732 14314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043: Generated new TSK 0
I20260619 04:36:28.538807 14314 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 04:36:28.593701 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41103
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:36:28.908908 14326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:28.909190 14326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:28.909314 14326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:28.916501 14326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:28.916610 14326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:28.916757 14326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:36:28.926574 14326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:28.927793 14326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:28.929111 14326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:28.941250 14331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:28.941270 14334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:28.941207 14332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:28.942047 14326 server_base.cc:1034] running on GCE node
I20260619 04:36:28.942569 14326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:28.943423 14326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:28.944594 14326 hybrid_clock.cc:648] HybridClock initialized: now 1781843788944547 us; error 36 us; skew 500 ppm
I20260619 04:36:28.944891 14326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:28 dist-test-slave-n3tq krb5kdc[14220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:28.947791 14326 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:36:28.949142 14326 webserver.cc:460] Webserver started at http://127.0.97.1:45233/ using document root <none> and password file <none>
I20260619 04:36:28.949779 14326 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:28.949887 14326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:28.950239 14326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:36:28.953084 14326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "3e25c6ffe3c744a69004e210f17fc498"
format_stamp: "Formatted at 2026-06-19 04:36:28 on dist-test-slave-n3tq"
server_key: "3121e5906846534d2d048cbf58c58d71"
server_key_iv: "a6ffcc1b0704663a204ee1bdb5fc9ac1"
server_key_version: "encryptionkey@0"
I20260619 04:36:28.953809 14326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3e25c6ffe3c744a69004e210f17fc498"
format_stamp: "Formatted at 2026-06-19 04:36:28 on dist-test-slave-n3tq"
server_key: "3121e5906846534d2d048cbf58c58d71"
server_key_iv: "a6ffcc1b0704663a204ee1bdb5fc9ac1"
server_key_version: "encryptionkey@0"
I20260619 04:36:28.958680 14326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:36:28.961966 14341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:28.963215 14326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:36:28.963357 14326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "3e25c6ffe3c744a69004e210f17fc498"
format_stamp: "Formatted at 2026-06-19 04:36:28 on dist-test-slave-n3tq"
server_key: "3121e5906846534d2d048cbf58c58d71"
server_key_iv: "a6ffcc1b0704663a204ee1bdb5fc9ac1"
server_key_version: "encryptionkey@0"
I20260619 04:36:28.963536 14326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:28.978255 14326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:28.982398 14326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:28.982652 14326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:28.983573 14326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:36:28.984983 14326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:36:28.985059 14326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:28.985186 14326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:36:28.985237 14326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:29.014705 14326 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38405
I20260619 04:36:29.014739 14453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38405 every 8 connection(s)
I20260619 04:36:29.015938 14326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:36:29.024945 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 14326
I20260619 04:36:29.025139 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:36:29.025761 388 external_mini_cluster.cc:1428] Setting key 1b0bcfba426c7967072ea69572efa75b
Jun 19 04:36:29 dist-test-slave-n3tq krb5kdc[14220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:36:29.035311 14456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:29.018477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55347 (local address 127.0.97.62:41103)
0619 04:36:29.018914 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:29.018919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:29.019825 (+ 906us) server_negotiation.cc:407] Connection header received
0619 04:36:29.021716 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:29.021722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:29.021812 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:29.021908 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:29.023418 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:29.024153 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:29.025775 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:29.026197 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:29.026313 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:29.029839 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:29.029844 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:29.029847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:29.029878 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:29.032328 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:29.033068 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:29.033072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:29.033074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:29.033199 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:29.033606 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:29.033609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:29.033611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:29.033885 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:29.034017 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:29.034707 (+ 690us) server_negotiation.cc:299] Negotiation successful
0619 04:36:29.034945 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":231,"threads_started":1}
I20260619 04:36:29.036248 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:29.018974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.1:55347)
0619 04:36:29.019674 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:29.019713 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:29.021440 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:29.022164 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:29.022172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:29.022680 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:29.023254 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:29.023272 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:29.024304 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:29.024308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:29.025639 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:29.025651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:29.026414 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:29.026422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:29.026650 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:29.027304 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:29.027325 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:29.029610 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:29.032500 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:29.032511 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:29.032530 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:29.032946 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:29.033300 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:29.033305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:29.033309 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:29.033494 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:29.034010 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:29.034018 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:29.034358 (+ 340us) client_negotiation.cc:769] Sending connection context
0619 04:36:29.034598 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 04:36:29.035128 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":269,"threads_started":1}
I20260619 04:36:29.037755 14454 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41103
I20260619 04:36:29.038107 14454 heartbeater.cc:464] Registering TS with master...
I20260619 04:36:29.038949 14454 heartbeater.cc:511] Master 127.0.97.62:41103 requested a full tablet report, sending...
I20260619 04:36:29.041072 14265 ts_manager.cc:194] Registered new tserver with Master: 3e25c6ffe3c744a69004e210f17fc498 (127.0.97.1:38405)
I20260619 04:36:29.042475 14265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55347
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 04:36:29.071592 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41103
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:36:29.385965 14460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:29.386255 14460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:29.386370 14460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:29.393906 14460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:29.394027 14460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:29.394176 14460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:36:29.404075 14460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:29.405364 14460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:29.406628 14460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:29.419008 14466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:29.419078 14465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:29.419059 14468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:29.419824 14460 server_base.cc:1034] running on GCE node
I20260619 04:36:29.420480 14460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:29.421412 14460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:29.422606 14460 hybrid_clock.cc:648] HybridClock initialized: now 1781843789422553 us; error 49 us; skew 500 ppm
I20260619 04:36:29.422904 14460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:29 dist-test-slave-n3tq krb5kdc[14220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:29.425873 14460 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:36:29.427280 14460 webserver.cc:460] Webserver started at http://127.0.97.2:34815/ using document root <none> and password file <none>
I20260619 04:36:29.427938 14460 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:29.428059 14460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:29.428332 14460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:36:29.431300 14460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "40bcd4c6d5ff49a795b78b2758d675f5"
format_stamp: "Formatted at 2026-06-19 04:36:29 on dist-test-slave-n3tq"
server_key: "5ddcf074483b99678223baeb00e08b37"
server_key_iv: "4ee4b7e5aad6090f90a7b04f0970fc32"
server_key_version: "encryptionkey@0"
I20260619 04:36:29.432060 14460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "40bcd4c6d5ff49a795b78b2758d675f5"
format_stamp: "Formatted at 2026-06-19 04:36:29 on dist-test-slave-n3tq"
server_key: "5ddcf074483b99678223baeb00e08b37"
server_key_iv: "4ee4b7e5aad6090f90a7b04f0970fc32"
server_key_version: "encryptionkey@0"
I20260619 04:36:29.437386 14460 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 04:36:29.440886 14475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:29.442217 14460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:36:29.442370 14460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "40bcd4c6d5ff49a795b78b2758d675f5"
format_stamp: "Formatted at 2026-06-19 04:36:29 on dist-test-slave-n3tq"
server_key: "5ddcf074483b99678223baeb00e08b37"
server_key_iv: "4ee4b7e5aad6090f90a7b04f0970fc32"
server_key_version: "encryptionkey@0"
I20260619 04:36:29.442546 14460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:29.464702 14460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:29.469872 14460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:29.470162 14460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:29.471288 14460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:36:29.472599 14460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:36:29.472672 14460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:29.472762 14460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:36:29.472803 14460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:29.501675 14460 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46457
I20260619 04:36:29.501724 14587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46457 every 8 connection(s)
I20260619 04:36:29.502908 14460 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:36:29.511439 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 14460
I20260619 04:36:29.511633 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:36:29.512192 388 external_mini_cluster.cc:1428] Setting key 77f6da5e6211b34da80990c12acaa11d
Jun 19 04:36:29 dist-test-slave-n3tq krb5kdc[14220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:36:29.522971 14456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:29.505281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51723 (local address 127.0.97.62:41103)
0619 04:36:29.505436 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:29.505440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:29.506575 (+ 1135us) server_negotiation.cc:407] Connection header received
0619 04:36:29.508371 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:29.508376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:29.508478 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:29.508567 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:29.510047 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:29.510777 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:29.513004 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:29.513467 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:29.513582 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:29.517276 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:29.517281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:29.517284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:29.517322 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:29.519839 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:29.520546 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:29.520552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:29.520556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:29.520689 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:29.521166 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:29.521170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:29.521172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:29.521449 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:29.521609 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:29.522388 (+ 779us) server_negotiation.cc:299] Negotiation successful
0619 04:36:29.522653 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 04:36:29.523742 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:29.505749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.2:51723)
0619 04:36:29.506423 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:29.506460 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:29.508104 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:29.508829 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:29.508837 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:29.509329 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:29.509886 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:29.509903 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:29.510918 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:29.510921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:29.512869 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:29.512882 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:29.513687 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:29.513692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:29.513888 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:29.514572 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:29.514608 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:29.517052 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:29.519972 (+ 2920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:29.519981 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:29.520000 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:29.520400 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:29.520792 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:29.520796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:29.520800 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:29.521034 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:29.521569 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:29.521577 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:29.522012 (+ 435us) client_negotiation.cc:769] Sending connection context
0619 04:36:29.522250 (+ 238us) client_negotiation.cc:240] Negotiation successful
0619 04:36:29.522655 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":268,"threads_started":1}
I20260619 04:36:29.525071 14588 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41103
I20260619 04:36:29.525364 14588 heartbeater.cc:464] Registering TS with master...
I20260619 04:36:29.526110 14588 heartbeater.cc:511] Master 127.0.97.62:41103 requested a full tablet report, sending...
I20260619 04:36:29.527559 14265 ts_manager.cc:194] Registered new tserver with Master: 40bcd4c6d5ff49a795b78b2758d675f5 (127.0.97.2:46457)
I20260619 04:36:29.528353 14265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51723
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 04:36:29.558728 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41103
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:36:29.874791 14593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:29.875064 14593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:29.875181 14593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:29.882294 14593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:29.882400 14593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:29.882516 14593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:36:29.892009 14593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:29.893316 14593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:29.894590 14593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:29.907042 14599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:29.907121 14598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:29.907022 14601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:29.907858 14593 server_base.cc:1034] running on GCE node
I20260619 04:36:29.908442 14593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:29.909415 14593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:29.910609 14593 hybrid_clock.cc:648] HybridClock initialized: now 1781843789910559 us; error 45 us; skew 500 ppm
I20260619 04:36:29.910900 14593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:29 dist-test-slave-n3tq krb5kdc[14220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:29.913864 14593 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:36:29.915202 14593 webserver.cc:460] Webserver started at http://127.0.97.3:44495/ using document root <none> and password file <none>
I20260619 04:36:29.915853 14593 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:29.915957 14593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:29.916239 14593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:36:29.919149 14593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "6ba2f932b2b84317bee8e5ab65234425"
format_stamp: "Formatted at 2026-06-19 04:36:29 on dist-test-slave-n3tq"
server_key: "f1046fe1d656c89a13208e406fd8c839"
server_key_iv: "af90ec6885612a9e56fd06bda43d1b28"
server_key_version: "encryptionkey@0"
I20260619 04:36:29.919880 14593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6ba2f932b2b84317bee8e5ab65234425"
format_stamp: "Formatted at 2026-06-19 04:36:29 on dist-test-slave-n3tq"
server_key: "f1046fe1d656c89a13208e406fd8c839"
server_key_iv: "af90ec6885612a9e56fd06bda43d1b28"
server_key_version: "encryptionkey@0"
I20260619 04:36:29.924763 14593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 04:36:29.928015 14608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:29.929275 14593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:36:29.929418 14593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "6ba2f932b2b84317bee8e5ab65234425"
format_stamp: "Formatted at 2026-06-19 04:36:29 on dist-test-slave-n3tq"
server_key: "f1046fe1d656c89a13208e406fd8c839"
server_key_iv: "af90ec6885612a9e56fd06bda43d1b28"
server_key_version: "encryptionkey@0"
I20260619 04:36:29.929602 14593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:29.965495 14593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:29.969451 14593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:29.969719 14593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:29.970643 14593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:36:29.971868 14593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:36:29.971942 14593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:29.972064 14593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:36:29.972115 14593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:30.001294 14593 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44013
I20260619 04:36:30.001338 14720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44013 every 8 connection(s)
I20260619 04:36:30.002545 14593 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:36:30.009603 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 14593
I20260619 04:36:30.009812 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:36:30.010308 388 external_mini_cluster.cc:1428] Setting key db2e45cbfc7ce2b0390aa46a45f2e213
Jun 19 04:36:30 dist-test-slave-n3tq krb5kdc[14220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:36:30.022946 14456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.005030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58899 (local address 127.0.97.62:41103)
0619 04:36:30.005211 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:30.005215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:30.006358 (+ 1143us) server_negotiation.cc:407] Connection header received
0619 04:36:30.008128 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:30.008134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:30.008220 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:30.008304 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:30.010106 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.010883 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.012897 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.013399 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.013497 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:30.017371 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:30.017376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:30.017380 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:30.017414 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:30.020008 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:30.020791 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:30.020795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:30.020798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:30.020916 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:30.021318 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:30.021322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:30.021324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:30.021597 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:30.021729 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:30.022393 (+ 664us) server_negotiation.cc:299] Negotiation successful
0619 04:36:30.022619 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 04:36:30.023864 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.005522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.3:58899)
0619 04:36:30.006201 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:30.006247 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:30.007860 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:30.008578 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:30.008586 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:30.009091 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:30.009909 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:30.009936 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.011041 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.011046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:30.012760 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:30.012773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.013592 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.013597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:30.013799 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:30.014513 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:30.014547 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:30.017162 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:30.020164 (+ 3002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:30.020173 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:30.020189 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:30.020670 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:30.021016 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:30.021021 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:30.021024 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:30.021205 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:30.021715 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:30.021723 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:30.022037 (+ 314us) client_negotiation.cc:769] Sending connection context
0619 04:36:30.022264 (+ 227us) client_negotiation.cc:240] Negotiation successful
0619 04:36:30.022789 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":272,"threads_started":1}
I20260619 04:36:30.025295 14721 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41103
I20260619 04:36:30.025614 14721 heartbeater.cc:464] Registering TS with master...
I20260619 04:36:30.026273 14721 heartbeater.cc:511] Master 127.0.97.62:41103 requested a full tablet report, sending...
I20260619 04:36:30.027637 14265 ts_manager.cc:194] Registered new tserver with Master: 6ba2f932b2b84317bee8e5ab65234425 (127.0.97.3:44013)
I20260619 04:36:30.028472 14265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58899
I20260619 04:36:30.037808 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:36:30.045863 14454 heartbeater.cc:503] Master 127.0.97.62:41103 was elected leader, sending a full tablet report...
I20260619 04:36:30.054111 14456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.042084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54114 (local address 127.0.97.62:41103)
0619 04:36:30.042318 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:30.042324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:30.042622 (+ 298us) server_negotiation.cc:407] Connection header received
0619 04:36:30.042755 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:30.042759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:30.042827 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:30.042910 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:30.044162 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.044976 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.046395 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.046944 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.047052 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:30.049470 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:30.049475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:30.049478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:30.049511 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:30.051777 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:30.052405 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:30.052409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:30.052411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:30.052570 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:30.052917 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:30.052921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:30.052923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:30.053235 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:30.053407 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:30.053650 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 04:36:30.053816 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 04:36:30.057977 14265 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:36:30.064778 14265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:36:30.099340 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.076623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46457 (local address 127.0.0.1:44920)
0619 04:36:30.089797 (+ 13174us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:30.089815 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:30.089942 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:30.091152 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:30.091160 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:30.091209 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:36:30.092375 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:30.092389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.094196 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.094201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:30.096684 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:30.096698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.098532 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.098538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:30.098699 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:30.098727 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 04:36:30.098809 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 04:36:30.098951 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13060,"mutex_wait_us":172,"spinlock_wait_cycles":6656,"thread_start_us":12487,"threads_started":1}
I20260619 04:36:30.099431 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.077121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44920 (local address 127.0.97.2:46457)
0619 04:36:30.090409 (+ 13288us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:30.090416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:30.090438 (+ 22us) server_negotiation.cc:407] Connection header received
0619 04:36:30.090523 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:30.090529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:30.090713 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:30.090915 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:36:30.092571 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.094040 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.096868 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.098320 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.098455 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:30.098660 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:30.098910 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 04:36:30.099091 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13150,"thread_start_us":13011,"threads_started":1}
I20260619 04:36:30.099668 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.076512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38405 (local address 127.0.0.1:34440)
0619 04:36:30.089587 (+ 13075us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:30.089652 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:30.089841 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:30.091153 (+ 1312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:30.091160 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:30.091201 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:36:30.091791 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:30.091805 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.093456 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.093462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:30.095981 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:30.095997 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.098767 (+ 2770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.098772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:30.098948 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:30.098996 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 04:36:30.099064 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 04:36:30.099199 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12718,"mutex_wait_us":291,"spinlock_wait_cycles":1408,"thread_start_us":14663,"threads_started":1}
I20260619 04:36:30.100144 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.076803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34440 (local address 127.0.97.1:38405)
0619 04:36:30.090295 (+ 13492us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:30.090304 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:30.090335 (+ 31us) server_negotiation.cc:407] Connection header received
0619 04:36:30.090443 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:30.090452 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:30.090714 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:30.091581 (+ 867us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:36:30.091967 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.093309 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.097172 (+ 3863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.098545 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.099362 (+ 817us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:30.099608 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:30.099680 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 04:36:30.099847 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13312,"thread_start_us":14053,"threads_started":1}
I20260619 04:36:30.100580 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.077896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39544 (local address 127.0.97.3:44013)
0619 04:36:30.089373 (+ 11477us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:30.089383 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:30.090973 (+ 1590us) server_negotiation.cc:407] Connection header received
0619 04:36:30.091095 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:30.091102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:30.091328 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:30.091513 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:36:30.092342 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.094020 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.096891 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.097921 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.098039 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:30.098294 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:30.100017 (+ 1723us) server_negotiation.cc:299] Negotiation successful
0619 04:36:30.100206 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11289,"thread_start_us":11107,"threads_started":1}
I20260619 04:36:30.099258 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.077222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44013 (local address 127.0.0.1:39544)
0619 04:36:30.090726 (+ 13504us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:30.090743 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:30.090853 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:30.091515 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:30.091520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:30.091551 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:36:30.091993 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:30.092010 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.094164 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.094169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:30.096690 (+ 2521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:30.096706 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.098119 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.098125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:30.098316 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:30.098365 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 04:36:30.098494 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 04:36:30.098699 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13381,"mutex_wait_us":76,"thread_start_us":12305,"threads_started":1}
I20260619 04:36:30.103816 14523 tablet_service.cc:1467] Processing CreateTablet for tablet 78e123b64edc4a29bf7ef5479fb5b00b (DEFAULT_TABLE table=test-table [id=150f218713ca43e884d4add9f6ea15a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:36:30.103813 14389 tablet_service.cc:1467] Processing CreateTablet for tablet 78e123b64edc4a29bf7ef5479fb5b00b (DEFAULT_TABLE table=test-table [id=150f218713ca43e884d4add9f6ea15a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:36:30.104046 14656 tablet_service.cc:1467] Processing CreateTablet for tablet 78e123b64edc4a29bf7ef5479fb5b00b (DEFAULT_TABLE table=test-table [id=150f218713ca43e884d4add9f6ea15a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:36:30.105094 14523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e123b64edc4a29bf7ef5479fb5b00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:36:30.105253 14389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e123b64edc4a29bf7ef5479fb5b00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:36:30.105253 14656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e123b64edc4a29bf7ef5479fb5b00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:36:30.116051 14739 tablet_bootstrap.cc:492] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5: Bootstrap starting.
I20260619 04:36:30.120227 14739 tablet_bootstrap.cc:654] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5: Neither blocks nor log segments found. Creating new log.
I20260619 04:36:30.120512 14740 tablet_bootstrap.cc:492] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498: Bootstrap starting.
I20260619 04:36:30.120883 14741 tablet_bootstrap.cc:492] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425: Bootstrap starting.
I20260619 04:36:30.121632 14739 log.cc:824] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5: Log is configured to *not* fsync() on all Append() calls
I20260619 04:36:30.123754 14740 tablet_bootstrap.cc:654] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498: Neither blocks nor log segments found. Creating new log.
I20260619 04:36:30.124013 14741 tablet_bootstrap.cc:654] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425: Neither blocks nor log segments found. Creating new log.
I20260619 04:36:30.124779 14740 log.cc:824] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498: Log is configured to *not* fsync() on all Append() calls
I20260619 04:36:30.124853 14739 tablet_bootstrap.cc:492] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5: No bootstrap required, opened a new log
I20260619 04:36:30.125186 14739 ts_tablet_manager.cc:1397] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260619 04:36:30.125188 14741 log.cc:824] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425: Log is configured to *not* fsync() on all Append() calls
I20260619 04:36:30.127506 14741 tablet_bootstrap.cc:492] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425: No bootstrap required, opened a new log
I20260619 04:36:30.127552 14740 tablet_bootstrap.cc:492] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498: No bootstrap required, opened a new log
I20260619 04:36:30.127740 14741 ts_tablet_manager.cc:1397] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260619 04:36:30.127782 14740 ts_tablet_manager.cc:1397] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260619 04:36:30.135319 14741 raft_consensus.cc:348] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } }
I20260619 04:36:30.135694 14741 raft_consensus.cc:374] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:36:30.135799 14741 raft_consensus.cc:729] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ba2f932b2b84317bee8e5ab65234425, State: Initialized, Role: FOLLOWER
I20260619 04:36:30.135514 14739 raft_consensus.cc:348] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } }
I20260619 04:36:30.135926 14739 raft_consensus.cc:374] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:36:30.136034 14739 raft_consensus.cc:729] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40bcd4c6d5ff49a795b78b2758d675f5, State: Initialized, Role: FOLLOWER
I20260619 04:36:30.136406 14741 consensus_queue.cc:260] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } }
I20260619 04:36:30.136720 14739 consensus_queue.cc:260] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } }
I20260619 04:36:30.137564 14721 heartbeater.cc:503] Master 127.0.97.62:41103 was elected leader, sending a full tablet report...
I20260619 04:36:30.138108 14741 ts_tablet_manager.cc:1428] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260619 04:36:30.137984 14740 raft_consensus.cc:348] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } }
I20260619 04:36:30.138387 14740 raft_consensus.cc:374] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:36:30.138485 14740 raft_consensus.cc:729] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e25c6ffe3c744a69004e210f17fc498, State: Initialized, Role: FOLLOWER
I20260619 04:36:30.139057 14740 consensus_queue.cc:260] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } }
I20260619 04:36:30.140942 14740 ts_tablet_manager.cc:1428] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 04:36:30.145510 14588 heartbeater.cc:503] Master 127.0.97.62:41103 was elected leader, sending a full tablet report...
I20260619 04:36:30.146222 14739 ts_tablet_manager.cc:1428] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5: Time spent starting tablet: real 0.021s user 0.011s sys 0.007s
W20260619 04:36:30.254122 14722 tablet.cc:2369] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:36:30.254753 14589 tablet.cc:2369] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:36:30.268180 14455 tablet.cc:2369] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:36:30.330518 14746 raft_consensus.cc:482] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:36:30.330796 14746 raft_consensus.cc:504] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } }
I20260619 04:36:30.332511 14746 leader_election.cc:288] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e25c6ffe3c744a69004e210f17fc498 (127.0.97.1:38405), 6ba2f932b2b84317bee8e5ab65234425 (127.0.97.3:44013)
I20260619 04:36:30.339203 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.333280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45161 (local address 127.0.97.3:44013)
0619 04:36:30.333475 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:30.333481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:30.333930 (+ 449us) server_negotiation.cc:407] Connection header received
0619 04:36:30.334174 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:30.334178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:30.334239 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:30.334311 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:36:30.334880 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.335606 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.337520 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.338288 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.338397 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:30.338499 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:30.338825 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 04:36:30.338950 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 04:36:30.339299 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.332951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60975 (local address 127.0.97.1:38405)
0619 04:36:30.333219 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:30.333226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:30.334085 (+ 859us) server_negotiation.cc:407] Connection header received
0619 04:36:30.334252 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:30.334256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:30.334317 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:30.334391 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:36:30.335046 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.335745 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.337522 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.338319 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.338492 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:30.338591 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:30.338969 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 04:36:30.339096 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 04:36:30.339372 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.332826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44013 (local address 127.0.97.2:45161)
0619 04:36:30.333787 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:30.333810 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:30.334021 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:30.334341 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:30.334345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:30.334368 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:36:30.334737 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:30.334747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.335742 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.335746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:30.337372 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:30.337384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.338488 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.338494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:30.338660 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:30.338695 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 04:36:30.338799 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 04:36:30.338928 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":47,"spinlock_wait_cycles":35712,"thread_start_us":585,"threads_started":1}
I20260619 04:36:30.339453 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.332869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38405 (local address 127.0.97.2:60975)
0619 04:36:30.333984 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:30.333999 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:30.334123 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:30.334481 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:30.334485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:30.334539 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:36:30.334894 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:30.334902 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.335888 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.335892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:30.337371 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:30.337383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.338642 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.338648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:30.338827 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:30.338855 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 04:36:30.338924 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 04:36:30.339099 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"mutex_wait_us":29,"spinlock_wait_cycles":1664,"thread_start_us":801,"threads_started":1}
I20260619 04:36:30.340245 14745 raft_consensus.cc:482] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:36:30.340487 14676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e123b64edc4a29bf7ef5479fb5b00b" candidate_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ba2f932b2b84317bee8e5ab65234425" is_pre_election: true
I20260619 04:36:30.340473 14409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e123b64edc4a29bf7ef5479fb5b00b" candidate_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e25c6ffe3c744a69004e210f17fc498" is_pre_election: true
I20260619 04:36:30.340565 14745 raft_consensus.cc:504] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } }
I20260619 04:36:30.341049 14409 raft_consensus.cc:2454] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40bcd4c6d5ff49a795b78b2758d675f5 in term 0.
I20260619 04:36:30.341325 14676 raft_consensus.cc:2454] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40bcd4c6d5ff49a795b78b2758d675f5 in term 0.
I20260619 04:36:30.341810 14476 leader_election.cc:302] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e25c6ffe3c744a69004e210f17fc498, 40bcd4c6d5ff49a795b78b2758d675f5; no voters:
I20260619 04:36:30.342222 14746 raft_consensus.cc:2781] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:36:30.342352 14746 raft_consensus.cc:482] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:36:30.342450 14746 raft_consensus.cc:3037] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:36:30.342475 14745 leader_election.cc:288] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e25c6ffe3c744a69004e210f17fc498 (127.0.97.1:38405), 40bcd4c6d5ff49a795b78b2758d675f5 (127.0.97.2:46457)
I20260619 04:36:30.345364 14746 raft_consensus.cc:504] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } }
I20260619 04:36:30.346467 14409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e123b64edc4a29bf7ef5479fb5b00b" candidate_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e25c6ffe3c744a69004e210f17fc498"
I20260619 04:36:30.346557 14676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e123b64edc4a29bf7ef5479fb5b00b" candidate_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ba2f932b2b84317bee8e5ab65234425"
I20260619 04:36:30.346674 14409 raft_consensus.cc:3037] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:36:30.346764 14676 raft_consensus.cc:3037] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:36:30.349668 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.342934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38405 (local address 127.0.97.3:36157)
0619 04:36:30.343098 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:30.343115 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:30.343215 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:30.343751 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:30.343756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:30.343785 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:36:30.343956 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:30.343966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.345222 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.345227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:30.347203 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:30.347217 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.348911 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.348918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:30.349134 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:30.349182 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 04:36:30.349247 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 04:36:30.349385 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 04:36:30.350098 14676 raft_consensus.cc:2454] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40bcd4c6d5ff49a795b78b2758d675f5 in term 1.
I20260619 04:36:30.350272 14409 raft_consensus.cc:2454] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40bcd4c6d5ff49a795b78b2758d675f5 in term 1.
I20260619 04:36:30.350539 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.342933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36157 (local address 127.0.97.1:38405)
0619 04:36:30.343440 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:30.343446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:30.343461 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:36:30.343518 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:30.343523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:30.343597 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:30.343725 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:36:30.344081 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.345108 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.347409 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.348743 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.349997 (+ 1254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:30.350106 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:30.350158 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 04:36:30.350290 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382}
I20260619 04:36:30.350554 14477 leader_election.cc:302] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40bcd4c6d5ff49a795b78b2758d675f5, 6ba2f932b2b84317bee8e5ab65234425; no voters:
I20260619 04:36:30.351166 14746 leader_election.cc:288] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5 [CANDIDATE]: Term 1 election: Requested vote from peers 3e25c6ffe3c744a69004e210f17fc498 (127.0.97.1:38405), 6ba2f932b2b84317bee8e5ab65234425 (127.0.97.3:44013)
I20260619 04:36:30.351111 14409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e123b64edc4a29bf7ef5479fb5b00b" candidate_uuid: "6ba2f932b2b84317bee8e5ab65234425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e25c6ffe3c744a69004e210f17fc498" is_pre_election: true
I20260619 04:36:30.351229 14751 raft_consensus.cc:2781] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:36:30.351440 14409 raft_consensus.cc:2379] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ba2f932b2b84317bee8e5ab65234425 in current term 1: Already voted for candidate 40bcd4c6d5ff49a795b78b2758d675f5 in this term.
I20260619 04:36:30.351774 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.343487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53261 (local address 127.0.97.2:46457)
0619 04:36:30.343648 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:30.343654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:30.345011 (+ 1357us) server_negotiation.cc:407] Connection header received
0619 04:36:30.345163 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:30.345169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:30.345239 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:30.345314 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:36:30.345961 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.346978 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.349485 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.350646 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.350756 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:30.350838 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:30.351366 (+ 528us) server_negotiation.cc:299] Negotiation successful
0619 04:36:30.351505 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:36:30.351873 14751 raft_consensus.cc:686] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5 [term 1 LEADER]: Becoming Leader. State: Replica: 40bcd4c6d5ff49a795b78b2758d675f5, State: Running, Role: LEADER
I20260619 04:36:30.352619 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.343332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46457 (local address 127.0.97.3:53261)
0619 04:36:30.344902 (+ 1570us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:30.344924 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:30.345048 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:30.345350 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:30.345354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:30.345379 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:36:30.345787 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:30.345798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.347183 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.347188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:30.349314 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:30.349329 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.351019 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.351025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:30.351182 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:30.351217 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 04:36:30.352200 (+ 983us) client_negotiation.cc:240] Negotiation successful
0619 04:36:30.352335 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1458,"thread_start_us":1282,"threads_started":1}
I20260619 04:36:30.352394 14751 consensus_queue.cc:237] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } }
I20260619 04:36:30.353482 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e123b64edc4a29bf7ef5479fb5b00b" candidate_uuid: "6ba2f932b2b84317bee8e5ab65234425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" is_pre_election: true
I20260619 04:36:30.354406 14612 leader_election.cc:302] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ba2f932b2b84317bee8e5ab65234425; no voters: 3e25c6ffe3c744a69004e210f17fc498, 40bcd4c6d5ff49a795b78b2758d675f5
I20260619 04:36:30.354781 14745 raft_consensus.cc:2726] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 04:36:30.357154 14263 catalog_manager.cc:5495] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40bcd4c6d5ff49a795b78b2758d675f5 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } health_report { overall_health: UNKNOWN } } }
I20260619 04:36:30.386332 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.381287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44928 (local address 127.0.97.2:46457)
0619 04:36:30.381480 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:30.381484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:30.381533 (+ 49us) server_negotiation.cc:407] Connection header received
0619 04:36:30.381770 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:30.381775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:30.381885 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:30.382018 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:36:30.382347 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.383121 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.384430 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.384847 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.384937 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:30.385387 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:36:30.385904 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:36:30.386011 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:30.386047 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 04:36:30.386113 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 04:36:30.394474 14409 raft_consensus.cc:1261] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Refusing update from remote peer 40bcd4c6d5ff49a795b78b2758d675f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:36:30.394513 14676 raft_consensus.cc:1261] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425 [term 1 FOLLOWER]: Refusing update from remote peer 40bcd4c6d5ff49a795b78b2758d675f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:36:30.395340 14752 consensus_queue.cc:1035] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:36:30.395668 14751 consensus_queue.cc:1035] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:36:30.419595 14263 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:36:30.419837 14263 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:36:30.423187 14263 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043: Sending DeleteTablet for 3 replicas of tablet 78e123b64edc4a29bf7ef5479fb5b00b
I20260619 04:36:30.424347 14389 tablet_service.cc:1514] Processing DeleteTablet for tablet 78e123b64edc4a29bf7ef5479fb5b00b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:36:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34440
I20260619 04:36:30.424338 14656 tablet_service.cc:1514] Processing DeleteTablet for tablet 78e123b64edc4a29bf7ef5479fb5b00b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:36:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39544
I20260619 04:36:30.424652 14523 tablet_service.cc:1514] Processing DeleteTablet for tablet 78e123b64edc4a29bf7ef5479fb5b00b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:36:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44920
I20260619 04:36:30.425606 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 14326
I20260619 04:36:30.437974 14766 tablet_replica.cc:330] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5: stopping tablet replica
I20260619 04:36:30.438613 14765 tablet_replica.cc:330] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425: stopping tablet replica
I20260619 04:36:30.439075 14765 raft_consensus.cc:2229] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:36:30.439639 14765 raft_consensus.cc:2258] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:36:30.440649 14766 raft_consensus.cc:2229] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:36:30.441304 14766 raft_consensus.cc:2258] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:36:30.442237 14765 ts_tablet_manager.cc:1905] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:36:30.443349 14766 ts_tablet_manager.cc:1905] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 04:36:30 dist-test-slave-n3tq krb5kdc[14220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 04:36:30.450385 14764 tablet_replica.cc:330] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498: stopping tablet replica
I20260619 04:36:30.451982 14765 ts_tablet_manager.cc:1918] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:36:30.452582 14764 raft_consensus.cc:2229] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:36:30.453166 14764 raft_consensus.cc:2258] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:36:30.453688 14765 log.cc:1196] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/78e123b64edc4a29bf7ef5479fb5b00b
I20260619 04:36:30.454250 14765 ts_tablet_manager.cc:1939] T 78e123b64edc4a29bf7ef5479fb5b00b P 6ba2f932b2b84317bee8e5ab65234425: Deleting consensus metadata
I20260619 04:36:30.455924 14764 ts_tablet_manager.cc:1905] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:36:30.456162 14250 catalog_manager.cc:4841] TS 6ba2f932b2b84317bee8e5ab65234425 (127.0.97.3:44013): tablet 78e123b64edc4a29bf7ef5479fb5b00b (table test-table [id=150f218713ca43e884d4add9f6ea15a4]) successfully deleted
I20260619 04:36:30.457228 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.426752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34452 (local address 127.0.97.1:38405)
0619 04:36:30.426926 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:30.426932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:30.433805 (+ 6873us) server_negotiation.cc:407] Connection header received
0619 04:36:30.434874 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:30.434881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:30.434993 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:30.435071 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:30.438220 (+ 3149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.439330 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.441484 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.442065 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.442162 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:30.447601 (+ 5439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:30.447622 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:30.447640 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:30.447697 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:30.451480 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:30.452310 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:30.452319 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:30.452322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:30.452571 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:30.453004 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:30.453009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:30.453013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:30.453484 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:30.453916 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:30.456680 (+ 2764us) server_negotiation.cc:299] Negotiation successful
0619 04:36:30.456910 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:36:30.458683 14449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:30.464046 14766 ts_tablet_manager.cc:1918] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:36:30.464226 14766 log.cc:1196] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/78e123b64edc4a29bf7ef5479fb5b00b
I20260619 04:36:30.464769 14766 ts_tablet_manager.cc:1939] T 78e123b64edc4a29bf7ef5479fb5b00b P 40bcd4c6d5ff49a795b78b2758d675f5: Deleting consensus metadata
I20260619 04:36:30.474419 14252 catalog_manager.cc:4841] TS 40bcd4c6d5ff49a795b78b2758d675f5 (127.0.97.2:46457): tablet 78e123b64edc4a29bf7ef5479fb5b00b (table test-table [id=150f218713ca43e884d4add9f6ea15a4]) successfully deleted
I20260619 04:36:30.555045 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 14326
W20260619 04:36:30.569803 14249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:38405 (error 108)
I20260619 04:36:30.571137 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 14460
I20260619 04:36:30.571465 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.571061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38405 (local address 127.0.0.1:34456)
0619 04:36:30.571233 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:30.571325 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 04:36:30.571662 14249 catalog_manager.cc:4584] TS 3e25c6ffe3c744a69004e210f17fc498 (127.0.97.1:38405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 78e123b64edc4a29bf7ef5479fb5b00b: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38405: connect: Connection refused (error 111)
Jun 19 04:36:30 dist-test-slave-n3tq krb5kdc[14220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 04:36:30.588091 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.572027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44942 (local address 127.0.97.2:46457)
0619 04:36:30.572169 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:30.572174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:30.572191 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:36:30.572305 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:30.572310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:30.572376 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:30.572487 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:30.574026 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.574775 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.576675 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.577122 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.577224 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:30.582160 (+ 4936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:30.582177 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:30.582193 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:30.582234 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:30.585372 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:30.586026 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:30.586033 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:30.586035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:30.586177 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:30.586533 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:30.586537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:30.586539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:30.586943 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:30.587226 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:30.587565 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 04:36:30.587786 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 04:36:30.588804 14583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:30.625382 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.624865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38405 (local address 127.0.0.1:34460)
0619 04:36:30.625086 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:30.625210 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 04:36:30.626374 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.626029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38405 (local address 127.0.0.1:34464)
0619 04:36:30.626161 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:30.626251 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 04:36:30.686779 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.686201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38405 (local address 127.0.0.1:34468)
0619 04:36:30.686441 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:30.686571 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 04:36:30.687772 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.687397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38405 (local address 127.0.0.1:34482)
0619 04:36:30.687556 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:30.687649 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 04:36:30.703415 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 14460
I20260619 04:36:30.718622 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 14593
Jun 19 04:36:30 dist-test-slave-n3tq krb5kdc[14220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 04:36:30.732901 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.719313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39560 (local address 127.0.97.3:44013)
0619 04:36:30.719484 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:30.719489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:30.719591 (+ 102us) server_negotiation.cc:407] Connection header received
0619 04:36:30.719747 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:30.719751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:30.719823 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:30.719894 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:30.721146 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.721889 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.723029 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:30.723391 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:30.723480 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:30.727678 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:30.727692 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:30.727708 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:30.727740 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:30.730307 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:30.730907 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:30.730915 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:30.730918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:30.731051 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:30.731393 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:30.731397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:30.731399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:30.731741 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:30.732013 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:30.732334 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 04:36:30.732569 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 04:36:30.733505 14716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:30.768625 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.768104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38405 (local address 127.0.0.1:34486)
0619 04:36:30.768309 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:30.768394 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 04:36:30.769605 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.769224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38405 (local address 127.0.0.1:34492)
0619 04:36:30.769405 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:30.769486 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 04:36:30.822082 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 14593
I20260619 04:36:30.835584 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 14233
I20260619 04:36:30.836733 14295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:30.947120 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:30.946641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38405 (local address 127.0.0.1:34506)
0619 04:36:30.946867 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:30.946948 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 04:36:30.947705 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 14233
I20260619 04:36:30.965147 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41103
--webserver_interface=127.0.97.62
--webserver_port=36555
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:36:31.284031 14773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:31.284314 14773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:31.284401 14773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:31.291675 14773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:36:31.291788 14773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:31.291839 14773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:31.291893 14773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:36:31.291939 14773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:36:31.302083 14773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:31.303310 14773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:31.304653 14773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:31.325333 14782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:31.325430 14780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:31.325356 14779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:31.326078 14773 server_base.cc:1034] running on GCE node
I20260619 04:36:31.326903 14773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:31.328490 14773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:31.329734 14773 hybrid_clock.cc:648] HybridClock initialized: now 1781843791329680 us; error 36 us; skew 500 ppm
I20260619 04:36:31.330149 14773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:31 dist-test-slave-n3tq krb5kdc[14220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:31.333576 14773 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:36:31.335170 14773 webserver.cc:460] Webserver started at http://127.0.97.62:36555/ using document root <none> and password file <none>
I20260619 04:36:31.335870 14773 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:31.335981 14773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:31.341914 14773 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 04:36:31.344822 14789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:31.346196 14773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:36:31.346342 14773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "d6c6cbf4cf7d4996840f70e43e689043"
format_stamp: "Formatted at 2026-06-19 04:36:28 on dist-test-slave-n3tq"
server_key: "492fe3b4eb8d5b9866b9e0d7c5ab9deb"
server_key_iv: "3c83bf2d423230d62bb83c209efacc9f"
server_key_version: "encryptionkey@0"
I20260619 04:36:31.347386 14773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:31.360615 14773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:31.364704 14773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:31.364954 14773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:31.382678 14773 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41103
I20260619 04:36:31.382699 14840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41103 every 8 connection(s)
I20260619 04:36:31.384195 14773 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:36:31.386044 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 14773
I20260619 04:36:31.386801 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:38405
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41103
--webserver_port=45233
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:36:31.391263 14841 sys_catalog.cc:263] Verifying existing consensus state
I20260619 04:36:31.394764 14841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043: Bootstrap starting.
I20260619 04:36:31.414091 14841 log.cc:824] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043: Log is configured to *not* fsync() on all Append() calls
I20260619 04:36:31.426242 14841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 04:36:31.426764 14841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043: Bootstrap complete.
I20260619 04:36:31.435647 14841 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c6cbf4cf7d4996840f70e43e689043" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41103 } }
I20260619 04:36:31.436492 14841 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c6cbf4cf7d4996840f70e43e689043, State: Initialized, Role: FOLLOWER
I20260619 04:36:31.437059 14841 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c6cbf4cf7d4996840f70e43e689043" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41103 } }
I20260619 04:36:31.437285 14841 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:36:31.437388 14841 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:36:31.437543 14841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 1 FOLLOWER]: Advancing to term 2
I20260619 04:36:31.441457 14841 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c6cbf4cf7d4996840f70e43e689043" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41103 } }
I20260619 04:36:31.441937 14841 leader_election.cc:302] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6c6cbf4cf7d4996840f70e43e689043; no voters:
I20260619 04:36:31.442504 14841 leader_election.cc:288] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 04:36:31.442631 14845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 04:36:31.443675 14845 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 2 LEADER]: Becoming Leader. State: Replica: d6c6cbf4cf7d4996840f70e43e689043, State: Running, Role: LEADER
I20260619 04:36:31.444667 14841 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:36:31.444239 14845 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c6cbf4cf7d4996840f70e43e689043" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41103 } }
I20260619 04:36:31.446920 14847 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6c6cbf4cf7d4996840f70e43e689043. Latest consensus state: current_term: 2 leader_uuid: "d6c6cbf4cf7d4996840f70e43e689043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c6cbf4cf7d4996840f70e43e689043" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41103 } } }
I20260619 04:36:31.447305 14847 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [sys.catalog]: This master's current role is: LEADER
I20260619 04:36:31.447319 14846 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d6c6cbf4cf7d4996840f70e43e689043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c6cbf4cf7d4996840f70e43e689043" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41103 } } }
I20260619 04:36:31.447909 14846 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [sys.catalog]: This master's current role is: LEADER
I20260619 04:36:31.461805 14852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:36:31.470247 14852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:36:31.471714 14852 catalog_manager.cc:1259] Loaded cluster ID: 8f56ce11127643ef9b1d883a073e62f6
I20260619 04:36:31.471823 14852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:36:31.474988 14852 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 04:36:31.494630 14852 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 04:36:31.504243 14852 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:36:31.505896 14852 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043: Loaded TSK: 0
I20260619 04:36:31.507432 14852 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 04:36:31.728120 14843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:31.728422 14843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:31.728569 14843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:31.735976 14843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:31.736088 14843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:31.736227 14843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:36:31.746541 14843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:38405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45233
--tserver_master_addrs=127.0.97.62:41103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:31.747857 14843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:31.749214 14843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:31.763860 14868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:31.763993 14869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:31.764516 14871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:31.764889 14843 server_base.cc:1034] running on GCE node
I20260619 04:36:31.765483 14843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:31.766492 14843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:31.767705 14843 hybrid_clock.cc:648] HybridClock initialized: now 1781843791767648 us; error 51 us; skew 500 ppm
I20260619 04:36:31.768007 14843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:31 dist-test-slave-n3tq krb5kdc[14220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:31.771409 14843 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:36:31.773274 14843 webserver.cc:460] Webserver started at http://127.0.97.1:45233/ using document root <none> and password file <none>
I20260619 04:36:31.774238 14843 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:31.774399 14843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:31.781844 14843 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 04:36:31.785633 14878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:31.787350 14843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:36:31.787528 14843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "3e25c6ffe3c744a69004e210f17fc498"
format_stamp: "Formatted at 2026-06-19 04:36:28 on dist-test-slave-n3tq"
server_key: "3121e5906846534d2d048cbf58c58d71"
server_key_iv: "a6ffcc1b0704663a204ee1bdb5fc9ac1"
server_key_version: "encryptionkey@0"
I20260619 04:36:31.788852 14843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:31.812918 14843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:31.818148 14843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:31.818430 14843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:31.819545 14843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:36:31.821656 14885 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260619 04:36:31.827057 14843 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 04:36:31.827150 14843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260619 04:36:31.827258 14843 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 04:36:31.830294 14843 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 04:36:31.830381 14843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.001s
I20260619 04:36:31.830806 14885 tablet_bootstrap.cc:492] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498: Bootstrap starting.
I20260619 04:36:31.840582 14885 log.cc:824] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498: Log is configured to *not* fsync() on all Append() calls
I20260619 04:36:31.857007 14885 tablet_bootstrap.cc:492] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 04:36:31.857858 14885 tablet_bootstrap.cc:492] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498: Bootstrap complete.
I20260619 04:36:31.858459 14885 ts_tablet_manager.cc:1397] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498: Time spent bootstrapping tablet: real 0.028s user 0.017s sys 0.008s
I20260619 04:36:31.864388 14843 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38405
I20260619 04:36:31.864413 14992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38405 every 8 connection(s)
I20260619 04:36:31.866129 14843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:36:31.868072 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 14843
I20260619 04:36:31.868898 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:46457
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41103
--webserver_port=34815
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:36:31.869228 14885 raft_consensus.cc:348] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } }
I20260619 04:36:31.870658 14885 raft_consensus.cc:729] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e25c6ffe3c744a69004e210f17fc498, State: Initialized, Role: FOLLOWER
I20260619 04:36:31.871336 14885 consensus_queue.cc:260] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } }
I20260619 04:36:31.877799 14885 ts_tablet_manager.cc:1428] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498: Time spent starting tablet: real 0.019s user 0.019s sys 0.001s
Jun 19 04:36:31 dist-test-slave-n3tq krb5kdc[14220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:36:31.896276 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:31.869561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44369 (local address 127.0.97.62:41103)
0619 04:36:31.876941 (+ 7380us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:31.876954 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:31.877003 (+ 49us) server_negotiation.cc:407] Connection header received
0619 04:36:31.877318 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:31.877332 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:31.877687 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:31.878031 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:31.880057 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:31.881360 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:31.883529 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:31.884244 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:31.884378 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:31.888280 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:31.888289 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:31.888304 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:31.888346 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:31.891407 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:31.892371 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:31.892379 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:31.892385 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:31.892597 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:31.893095 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:31.893100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:31.893103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:31.893657 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:31.893812 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:31.894825 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0619 04:36:31.895338 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7171,"thread_start_us":7040,"threads_started":1}
I20260619 04:36:31.896517 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:31.869539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.1:44369)
0619 04:36:31.873537 (+ 3998us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:31.873586 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:31.874136 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:31.878200 (+ 4064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:31.878213 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:31.879113 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:31.879810 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:31.879829 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:31.881537 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:31.881542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:31.883336 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:31.883351 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:31.884421 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:31.884428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:31.884737 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:31.885428 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:31.885454 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:31.888013 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:31.891577 (+ 3564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:31.891586 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:31.891603 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:31.892226 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:31.892739 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:31.892744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:31.892748 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:31.892973 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:31.893808 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:31.893816 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:31.894282 (+ 466us) client_negotiation.cc:769] Sending connection context
0619 04:36:31.894653 (+ 371us) client_negotiation.cc:240] Negotiation successful
0619 04:36:31.895235 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3665,"thread_start_us":3438,"threads_started":1}
I20260619 04:36:31.899540 14993 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41103
I20260619 04:36:31.899837 14993 heartbeater.cc:464] Registering TS with master...
I20260619 04:36:31.900542 14993 heartbeater.cc:511] Master 127.0.97.62:41103 requested a full tablet report, sending...
I20260619 04:36:31.903795 14806 ts_manager.cc:194] Registered new tserver with Master: 3e25c6ffe3c744a69004e210f17fc498 (127.0.97.1:38405)
I20260619 04:36:31.908717 14806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44369
I20260619 04:36:31.912681 14993 heartbeater.cc:503] Master 127.0.97.62:41103 was elected leader, sending a full tablet report...
I20260619 04:36:31.923277 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:31.907858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38405 (local address 127.0.0.1:34510)
0619 04:36:31.916085 (+ 8227us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:31.916140 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:31.916325 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:31.917367 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:31.917372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:31.917395 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:36:31.917661 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:31.917670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:31.919417 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:31.919421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:31.920956 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:31.920968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:31.922629 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:31.922634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:31.922766 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:31.922798 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 04:36:31.922894 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 04:36:31.923020 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7935,"thread_start_us":7750,"threads_started":1}
I20260619 04:36:31.923692 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:31.907928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34510 (local address 127.0.97.1:38405)
0619 04:36:31.916846 (+ 8918us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:31.916854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:31.916883 (+ 29us) server_negotiation.cc:407] Connection header received
0619 04:36:31.916963 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:31.916970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:31.917175 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:31.917351 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:36:31.917809 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:31.919301 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:31.921107 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:31.922442 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:31.922573 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:31.922807 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:31.922931 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 04:36:31.923077 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8762,"thread_start_us":8636,"threads_started":1}
I20260619 04:36:31.924850 14928 tablet_service.cc:1514] Processing DeleteTablet for tablet 78e123b64edc4a29bf7ef5479fb5b00b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:36:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34510
I20260619 04:36:31.926774 15001 tablet_replica.cc:330] stopping tablet replica
I20260619 04:36:31.927109 15001 raft_consensus.cc:2229] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:36:31.927456 15001 raft_consensus.cc:2258] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:36:31.928660 15001 ts_tablet_manager.cc:1905] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:36:31.935331 15001 ts_tablet_manager.cc:1918] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:36:31.935464 15001 log.cc:1196] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/wal/wals/78e123b64edc4a29bf7ef5479fb5b00b
I20260619 04:36:31.935848 15001 ts_tablet_manager.cc:1939] T 78e123b64edc4a29bf7ef5479fb5b00b P 3e25c6ffe3c744a69004e210f17fc498: Deleting consensus metadata
I20260619 04:36:31.937808 14790 catalog_manager.cc:4841] TS 3e25c6ffe3c744a69004e210f17fc498 (127.0.97.1:38405): tablet 78e123b64edc4a29bf7ef5479fb5b00b (table test-table [id=150f218713ca43e884d4add9f6ea15a4]) successfully deleted
W20260619 04:36:32.195506 14998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:32.195835 14998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:32.195986 14998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:32.203192 14998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:32.203310 14998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:32.203475 14998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:36:32.213132 14998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:46457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34815
--tserver_master_addrs=127.0.97.62:41103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:32.214412 14998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:32.215665 14998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:32.227916 15006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:32.227969 15009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:32.227950 15007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:32.229072 14998 server_base.cc:1034] running on GCE node
I20260619 04:36:32.229619 14998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:32.230497 14998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:32.231710 14998 hybrid_clock.cc:648] HybridClock initialized: now 1781843792231655 us; error 46 us; skew 500 ppm
I20260619 04:36:32.232023 14998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:32 dist-test-slave-n3tq krb5kdc[14220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:32.235090 14998 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:36:32.236512 14998 webserver.cc:460] Webserver started at http://127.0.97.2:34815/ using document root <none> and password file <none>
I20260619 04:36:32.237174 14998 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:32.237299 14998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:32.242632 14998 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 04:36:32.245358 15016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:32.246755 14998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:36:32.246905 14998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "40bcd4c6d5ff49a795b78b2758d675f5"
format_stamp: "Formatted at 2026-06-19 04:36:29 on dist-test-slave-n3tq"
server_key: "5ddcf074483b99678223baeb00e08b37"
server_key_iv: "4ee4b7e5aad6090f90a7b04f0970fc32"
server_key_version: "encryptionkey@0"
I20260619 04:36:32.247849 14998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:32.270738 14998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:32.274899 14998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:32.275174 14998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:32.276163 14998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:36:32.277618 14998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:36:32.277724 14998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:32.277837 14998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:36:32.277911 14998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:32.308943 14998 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46457
I20260619 04:36:32.309010 15128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46457 every 8 connection(s)
I20260619 04:36:32.310345 14998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:36:32.316250 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 14998
I20260619 04:36:32.316954 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44013
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41103
--webserver_port=44495
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:36:32 dist-test-slave-n3tq krb5kdc[14220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:36:32.330729 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:32.312935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43607 (local address 127.0.97.62:41103)
0619 04:36:32.313104 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:32.313108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:32.314289 (+ 1181us) server_negotiation.cc:407] Connection header received
0619 04:36:32.316089 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:32.316095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:32.316186 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:32.316306 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:32.317925 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.318903 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.320634 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.321253 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.321388 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:32.325098 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:32.325105 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:32.325109 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:32.325146 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:32.327605 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:32.328359 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:32.328363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:32.328366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:32.328525 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:32.328961 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:32.328965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:32.328967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:32.329247 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:32.329397 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:32.330135 (+ 738us) server_negotiation.cc:299] Negotiation successful
0619 04:36:32.330381 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 04:36:32.331804 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:32.313362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.2:43607)
0619 04:36:32.314115 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:32.314155 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:32.315780 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:32.316567 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:32.316576 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:32.317148 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:32.317732 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:32.317749 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.319111 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.319135 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:32.320393 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:32.320405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.321528 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.321536 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:32.321783 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:32.322435 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:32.322462 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:32.324853 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:32.327750 (+ 2897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:32.327760 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:32.327780 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:32.328222 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:32.328625 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:32.328630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:32.328634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:32.328840 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:32.329376 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:32.329384 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:32.329745 (+ 361us) client_negotiation.cc:769] Sending connection context
0619 04:36:32.330000 (+ 255us) client_negotiation.cc:240] Negotiation successful
0619 04:36:32.330562 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":306,"threads_started":1}
I20260619 04:36:32.333374 15129 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41103
I20260619 04:36:32.333693 15129 heartbeater.cc:464] Registering TS with master...
I20260619 04:36:32.334412 15129 heartbeater.cc:511] Master 127.0.97.62:41103 requested a full tablet report, sending...
I20260619 04:36:32.335912 14806 ts_manager.cc:194] Registered new tserver with Master: 40bcd4c6d5ff49a795b78b2758d675f5 (127.0.97.2:46457)
I20260619 04:36:32.337107 14806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43607
W20260619 04:36:32.635483 15132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:32.635773 15132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:32.635864 15132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:32.643513 15132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:32.643630 15132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:32.643743 15132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:36:32.653672 15132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44495
--tserver_master_addrs=127.0.97.62:41103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:32.654907 15132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:32.656167 15132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:32.668888 15137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:32.668819 15138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:32.668857 15140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:32.669697 15132 server_base.cc:1034] running on GCE node
I20260619 04:36:32.670310 15132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:32.671224 15132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:32.672426 15132 hybrid_clock.cc:648] HybridClock initialized: now 1781843792672376 us; error 43 us; skew 500 ppm
I20260619 04:36:32.672792 15132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:32 dist-test-slave-n3tq krb5kdc[14220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:32.675716 15132 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:36:32.677176 15132 webserver.cc:460] Webserver started at http://127.0.97.3:44495/ using document root <none> and password file <none>
I20260619 04:36:32.677831 15132 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:32.677955 15132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:32.683135 15132 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 04:36:32.685985 15147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:32.687340 15132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:36:32.687470 15132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "6ba2f932b2b84317bee8e5ab65234425"
format_stamp: "Formatted at 2026-06-19 04:36:29 on dist-test-slave-n3tq"
server_key: "f1046fe1d656c89a13208e406fd8c839"
server_key_iv: "af90ec6885612a9e56fd06bda43d1b28"
server_key_version: "encryptionkey@0"
I20260619 04:36:32.688409 15132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:32.714696 15132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:32.718629 15132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:32.718851 15132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:32.719772 15132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:36:32.721153 15132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:36:32.721235 15132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:32.721309 15132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:36:32.721360 15132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:32.751093 15132 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44013
I20260619 04:36:32.751178 15259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44013 every 8 connection(s)
I20260619 04:36:32.752592 15132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:36:32.758771 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 15132
Jun 19 04:36:32 dist-test-slave-n3tq krb5kdc[14220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:36:32.772265 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:32.759629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54124 (local address 127.0.97.62:41103)
0619 04:36:32.760311 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:32.760317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:32.760333 (+ 16us) server_negotiation.cc:407] Connection header received
0619 04:36:32.760382 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:32.760387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:32.760443 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:32.760590 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:32.762101 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.762833 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.764143 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.764588 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.764698 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:32.767113 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:32.767118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:32.767121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:32.767161 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:32.769745 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:32.770400 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:32.770406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:32.770409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:32.770515 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:32.770854 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:32.770859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:32.770863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:32.771114 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:32.771324 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:32.771645 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 04:36:32.771861 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":463,"threads_started":1}
I20260619 04:36:32.772383 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:32.755023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36657 (local address 127.0.97.62:41103)
0619 04:36:32.755200 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:32.755204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:32.756348 (+ 1144us) server_negotiation.cc:407] Connection header received
0619 04:36:32.758210 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:32.758215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:32.758300 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:32.758387 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:32.760184 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.761081 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.762614 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.763042 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.763146 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:32.766588 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:32.766593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:32.766596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:32.766626 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:32.769438 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:32.770144 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:32.770149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:32.770151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:32.770275 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:32.770682 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:32.770687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:32.770690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:32.770969 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:32.771112 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:32.771799 (+ 687us) server_negotiation.cc:299] Negotiation successful
0619 04:36:32.771996 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:36:32.773496 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:32.755507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.3:36657)
0619 04:36:32.756183 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:32.756229 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:32.757936 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:32.758660 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:32.758668 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:32.759217 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:32.759963 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:32.759993 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.761220 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.761224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:32.762483 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:32.762495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.763250 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.763258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:32.763471 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:32.764077 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:32.764099 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:32.766375 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:32.769589 (+ 3214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:32.769598 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:32.769613 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:32.770029 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:32.770383 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:32.770388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:32.770391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:32.770579 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:32.771091 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:32.771098 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:32.771432 (+ 334us) client_negotiation.cc:769] Sending connection context
0619 04:36:32.771683 (+ 251us) client_negotiation.cc:240] Negotiation successful
0619 04:36:32.772220 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":256,"threads_started":1}
I20260619 04:36:32.774860 15260 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41103
I20260619 04:36:32.775143 15260 heartbeater.cc:464] Registering TS with master...
I20260619 04:36:32.775808 15260 heartbeater.cc:511] Master 127.0.97.62:41103 requested a full tablet report, sending...
I20260619 04:36:32.777137 14806 ts_manager.cc:194] Registered new tserver with Master: 6ba2f932b2b84317bee8e5ab65234425 (127.0.97.3:44013)
I20260619 04:36:32.777902 14806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36657
I20260619 04:36:32.786962 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:36:32.793701 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:32.787823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54140 (local address 127.0.97.62:41103)
0619 04:36:32.788043 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:32.788051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:32.788437 (+ 386us) server_negotiation.cc:407] Connection header received
0619 04:36:32.788699 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:32.788705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:32.788845 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:32.788965 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:36:32.789328 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.790040 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.791395 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.791858 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.791949 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:32.792429 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:36:32.793115 (+ 686us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:36:32.793274 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:32.793351 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 04:36:32.793422 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 04:36:32.794665 14806 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:36:32.801745 14806 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:36:32.812963 14928 tablet_service.cc:1467] Processing CreateTablet for tablet 9c21326af9ff4eb991716ab8c07f8792 (DEFAULT_TABLE table=test-table [id=c416c502fca046519d83fbd8e34270d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:36:32.813927 14928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c21326af9ff4eb991716ab8c07f8792. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:36:32.825119 15274 tablet_bootstrap.cc:492] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498: Bootstrap starting.
I20260619 04:36:32.827061 15274 tablet_bootstrap.cc:654] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498: Neither blocks nor log segments found. Creating new log.
I20260619 04:36:32.830050 15274 tablet_bootstrap.cc:492] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498: No bootstrap required, opened a new log
I20260619 04:36:32.830194 15274 ts_tablet_manager.cc:1397] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 04:36:32.830964 15274 raft_consensus.cc:348] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } }
I20260619 04:36:32.831226 15274 raft_consensus.cc:374] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:36:32.831326 15274 raft_consensus.cc:729] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e25c6ffe3c744a69004e210f17fc498, State: Initialized, Role: FOLLOWER
I20260619 04:36:32.831560 15274 consensus_queue.cc:260] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } }
I20260619 04:36:32.832372 15274 ts_tablet_manager.cc:1428] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 04:36:32.836177 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:32.811765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39566 (local address 127.0.97.3:44013)
0619 04:36:32.826999 (+ 15234us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:32.827008 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:32.827050 (+ 42us) server_negotiation.cc:407] Connection header received
0619 04:36:32.827172 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:32.827181 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:32.827485 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:32.827730 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:36:32.828315 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.830371 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.833766 (+ 3395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.835191 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.835302 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:32.835537 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:32.835707 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 04:36:32.835875 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15041,"thread_start_us":16184,"threads_started":1}
I20260619 04:36:32.836665 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:32.811530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44013 (local address 127.0.0.1:39566)
0619 04:36:32.822540 (+ 11010us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:32.822571 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:32.822750 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:32.827720 (+ 4970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:32.827725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:32.827757 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:36:32.828104 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:32.828119 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.830511 (+ 2392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.830516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:32.833595 (+ 3079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:32.833610 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.835362 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.835368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:32.835526 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:32.835561 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 04:36:32.835658 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 04:36:32.835796 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10836,"mutex_wait_us":1,"thread_start_us":17577,"threads_started":1}
I20260619 04:36:32.837801 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:32.811851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46457 (local address 127.0.0.1:44956)
0619 04:36:32.829387 (+ 17536us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:32.829407 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:32.829620 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:32.830506 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:32.830511 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:32.830535 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:36:32.830844 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:32.830852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.832921 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.832944 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:32.835160 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:32.835176 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.837133 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.837139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:32.837263 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:32.837284 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 04:36:32.837361 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 04:36:32.837481 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17405,"thread_start_us":10360,"threads_started":1}
I20260619 04:36:32.837857 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:32.812530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44956 (local address 127.0.97.2:46457)
0619 04:36:32.828625 (+ 16095us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:32.828635 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:32.829503 (+ 868us) server_negotiation.cc:407] Connection header received
0619 04:36:32.829816 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:32.829824 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:32.830062 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:32.830209 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:36:32.831010 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.832801 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.835358 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.836815 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.836937 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:32.837131 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:32.837426 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 04:36:32.837582 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15854,"thread_start_us":16097,"threads_started":1}
I20260619 04:36:32.840641 15195 tablet_service.cc:1467] Processing CreateTablet for tablet 9c21326af9ff4eb991716ab8c07f8792 (DEFAULT_TABLE table=test-table [id=c416c502fca046519d83fbd8e34270d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:36:32.841137 15064 tablet_service.cc:1467] Processing CreateTablet for tablet 9c21326af9ff4eb991716ab8c07f8792 (DEFAULT_TABLE table=test-table [id=c416c502fca046519d83fbd8e34270d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:36:32.842123 15195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c21326af9ff4eb991716ab8c07f8792. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:36:32.842381 15064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c21326af9ff4eb991716ab8c07f8792. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:36:32.857528 15277 tablet_bootstrap.cc:492] T 9c21326af9ff4eb991716ab8c07f8792 P 40bcd4c6d5ff49a795b78b2758d675f5: Bootstrap starting.
I20260619 04:36:32.858435 15278 tablet_bootstrap.cc:492] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425: Bootstrap starting.
I20260619 04:36:32.861850 15277 tablet_bootstrap.cc:654] T 9c21326af9ff4eb991716ab8c07f8792 P 40bcd4c6d5ff49a795b78b2758d675f5: Neither blocks nor log segments found. Creating new log.
I20260619 04:36:32.862247 15278 tablet_bootstrap.cc:654] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425: Neither blocks nor log segments found. Creating new log.
I20260619 04:36:32.863214 15277 log.cc:824] T 9c21326af9ff4eb991716ab8c07f8792 P 40bcd4c6d5ff49a795b78b2758d675f5: Log is configured to *not* fsync() on all Append() calls
I20260619 04:36:32.863498 15278 log.cc:824] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425: Log is configured to *not* fsync() on all Append() calls
I20260619 04:36:32.865937 15277 tablet_bootstrap.cc:492] T 9c21326af9ff4eb991716ab8c07f8792 P 40bcd4c6d5ff49a795b78b2758d675f5: No bootstrap required, opened a new log
I20260619 04:36:32.866194 15277 ts_tablet_manager.cc:1397] T 9c21326af9ff4eb991716ab8c07f8792 P 40bcd4c6d5ff49a795b78b2758d675f5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 04:36:32.866357 15278 tablet_bootstrap.cc:492] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425: No bootstrap required, opened a new log
I20260619 04:36:32.866596 15278 ts_tablet_manager.cc:1397] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
W20260619 04:36:32.868341 14994 tablet.cc:2369] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:36:32.873596 15277 raft_consensus.cc:348] T 9c21326af9ff4eb991716ab8c07f8792 P 40bcd4c6d5ff49a795b78b2758d675f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } }
I20260619 04:36:32.873953 15277 raft_consensus.cc:374] T 9c21326af9ff4eb991716ab8c07f8792 P 40bcd4c6d5ff49a795b78b2758d675f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:36:32.874055 15277 raft_consensus.cc:729] T 9c21326af9ff4eb991716ab8c07f8792 P 40bcd4c6d5ff49a795b78b2758d675f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40bcd4c6d5ff49a795b78b2758d675f5, State: Initialized, Role: FOLLOWER
I20260619 04:36:32.874629 15277 consensus_queue.cc:260] T 9c21326af9ff4eb991716ab8c07f8792 P 40bcd4c6d5ff49a795b78b2758d675f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } }
I20260619 04:36:32.875687 15129 heartbeater.cc:503] Master 127.0.97.62:41103 was elected leader, sending a full tablet report...
I20260619 04:36:32.876343 15277 ts_tablet_manager.cc:1428] T 9c21326af9ff4eb991716ab8c07f8792 P 40bcd4c6d5ff49a795b78b2758d675f5: Time spent starting tablet: real 0.010s user 0.005s sys 0.006s
I20260619 04:36:32.877653 15278 raft_consensus.cc:348] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } }
I20260619 04:36:32.878047 15278 raft_consensus.cc:374] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:36:32.878144 15278 raft_consensus.cc:729] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ba2f932b2b84317bee8e5ab65234425, State: Initialized, Role: FOLLOWER
I20260619 04:36:32.878804 15278 consensus_queue.cc:260] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } }
I20260619 04:36:32.879863 15260 heartbeater.cc:503] Master 127.0.97.62:41103 was elected leader, sending a full tablet report...
I20260619 04:36:32.880447 15278 ts_tablet_manager.cc:1428] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260619 04:36:32.965342 15282 raft_consensus.cc:482] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:36:32.965648 15282 raft_consensus.cc:504] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } }
I20260619 04:36:32.967271 15282 leader_election.cc:288] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40bcd4c6d5ff49a795b78b2758d675f5 (127.0.97.2:46457), 3e25c6ffe3c744a69004e210f17fc498 (127.0.97.1:38405)
I20260619 04:36:32.974376 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:32.967875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35995 (local address 127.0.97.2:46457)
0619 04:36:32.968073 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:32.968079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:32.968102 (+ 23us) server_negotiation.cc:407] Connection header received
0619 04:36:32.968317 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:32.968322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:32.968399 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:32.968516 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:36:32.969079 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.970054 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.972378 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.973530 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.973666 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:32.973764 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:32.974010 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 04:36:32.974151 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 04:36:32.974380 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:32.967726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46457 (local address 127.0.97.3:35995)
0619 04:36:32.967913 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:32.967936 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:32.968105 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:32.968707 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:32.968712 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:32.968740 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:36:32.968941 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:32.968954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.970251 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.970256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:32.972229 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:32.972244 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.973697 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.973702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:32.973861 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:32.973886 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 04:36:32.973966 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 04:36:32.974071 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":1}
I20260619 04:36:32.974918 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:32.967742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38405 (local address 127.0.97.3:57183)
0619 04:36:32.968835 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:32.968856 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:32.968970 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:32.969289 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:32.969292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:32.969309 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:36:32.969576 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:32.969583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.970719 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.970725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:32.972665 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:32.972675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.974170 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.974176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:32.974336 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:32.974370 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 04:36:32.974462 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 04:36:32.974573 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":194,"thread_start_us":557,"threads_started":1}
I20260619 04:36:32.975024 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:32.967717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57183 (local address 127.0.97.1:38405)
0619 04:36:32.968572 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:32.968577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:32.968937 (+ 360us) server_negotiation.cc:407] Connection header received
0619 04:36:32.969079 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:32.969084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:32.969162 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:32.969267 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:36:32.969715 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.970554 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.972789 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:32.973974 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:32.974091 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:32.974195 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:32.974633 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 04:36:32.974749 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":542,"threads_started":1}
I20260619 04:36:32.975692 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c21326af9ff4eb991716ab8c07f8792" candidate_uuid: "6ba2f932b2b84317bee8e5ab65234425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" is_pre_election: true
I20260619 04:36:32.975852 14948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c21326af9ff4eb991716ab8c07f8792" candidate_uuid: "6ba2f932b2b84317bee8e5ab65234425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e25c6ffe3c744a69004e210f17fc498" is_pre_election: true
I20260619 04:36:32.976145 15084 raft_consensus.cc:2454] T 9c21326af9ff4eb991716ab8c07f8792 P 40bcd4c6d5ff49a795b78b2758d675f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ba2f932b2b84317bee8e5ab65234425 in term 0.
I20260619 04:36:32.976333 14948 raft_consensus.cc:2454] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ba2f932b2b84317bee8e5ab65234425 in term 0.
I20260619 04:36:32.976823 15151 leader_election.cc:302] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40bcd4c6d5ff49a795b78b2758d675f5, 6ba2f932b2b84317bee8e5ab65234425; no voters:
I20260619 04:36:32.977190 15282 raft_consensus.cc:2781] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:36:32.977332 15282 raft_consensus.cc:482] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:36:32.977427 15282 raft_consensus.cc:3037] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:36:32.979867 15282 raft_consensus.cc:504] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } }
I20260619 04:36:32.980429 15282 leader_election.cc:288] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425 [CANDIDATE]: Term 1 election: Requested vote from peers 40bcd4c6d5ff49a795b78b2758d675f5 (127.0.97.2:46457), 3e25c6ffe3c744a69004e210f17fc498 (127.0.97.1:38405)
I20260619 04:36:32.981020 14948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c21326af9ff4eb991716ab8c07f8792" candidate_uuid: "6ba2f932b2b84317bee8e5ab65234425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e25c6ffe3c744a69004e210f17fc498"
I20260619 04:36:32.981089 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c21326af9ff4eb991716ab8c07f8792" candidate_uuid: "6ba2f932b2b84317bee8e5ab65234425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40bcd4c6d5ff49a795b78b2758d675f5"
I20260619 04:36:32.981222 14948 raft_consensus.cc:3037] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:36:32.981319 15084 raft_consensus.cc:3037] T 9c21326af9ff4eb991716ab8c07f8792 P 40bcd4c6d5ff49a795b78b2758d675f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:36:32.983484 14948 raft_consensus.cc:2454] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ba2f932b2b84317bee8e5ab65234425 in term 1.
I20260619 04:36:32.983513 15084 raft_consensus.cc:2454] T 9c21326af9ff4eb991716ab8c07f8792 P 40bcd4c6d5ff49a795b78b2758d675f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ba2f932b2b84317bee8e5ab65234425 in term 1.
I20260619 04:36:32.983848 15148 leader_election.cc:302] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e25c6ffe3c744a69004e210f17fc498, 6ba2f932b2b84317bee8e5ab65234425; no voters:
I20260619 04:36:32.984148 15282 raft_consensus.cc:2781] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:36:32.984620 15282 raft_consensus.cc:686] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425 [term 1 LEADER]: Becoming Leader. State: Replica: 6ba2f932b2b84317bee8e5ab65234425, State: Running, Role: LEADER
I20260619 04:36:32.985071 15282 consensus_queue.cc:237] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } }
I20260619 04:36:32.989719 14805 catalog_manager.cc:5495] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ba2f932b2b84317bee8e5ab65234425 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6ba2f932b2b84317bee8e5ab65234425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } health_report { overall_health: UNKNOWN } } }
W20260619 04:36:33.004061 15261 tablet.cc:2369] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:36:33.030375 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:33.025462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39568 (local address 127.0.97.3:44013)
0619 04:36:33.025672 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:33.025678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:33.025697 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:36:33.025796 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:33.025800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:33.025874 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:33.025999 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:36:33.026280 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:33.026971 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:33.028288 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:33.028711 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:33.028796 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:33.029224 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:36:33.029724 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:36:33.029832 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:33.030016 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 04:36:33.030089 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 04:36:33.038307 14948 raft_consensus.cc:1261] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Refusing update from remote peer 6ba2f932b2b84317bee8e5ab65234425: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:36:33.038311 15084 raft_consensus.cc:1261] T 9c21326af9ff4eb991716ab8c07f8792 P 40bcd4c6d5ff49a795b78b2758d675f5 [term 1 FOLLOWER]: Refusing update from remote peer 6ba2f932b2b84317bee8e5ab65234425: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:36:33.039201 15285 consensus_queue.cc:1035] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:36:33.039585 15282 consensus_queue.cc:1035] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:36:33.050779 15288 mvcc.cc:205] Tried to move back new op lower bound from 7298432176275230720 to 7298432176070455296. Current Snapshot: MvccSnapshot[applied={T|T < 7298432176275230720}]
I20260619 04:36:33.062090 14806 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:36:33.062363 14806 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:36:33.065934 14806 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043: Sending DeleteTablet for 3 replicas of tablet 9c21326af9ff4eb991716ab8c07f8792
I20260619 04:36:33.066918 15195 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c21326af9ff4eb991716ab8c07f8792 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:36:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39566
I20260619 04:36:33.066946 15064 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c21326af9ff4eb991716ab8c07f8792 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:36:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44956
I20260619 04:36:33.067507 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 14843
I20260619 04:36:33.067595 14928 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c21326af9ff4eb991716ab8c07f8792 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:36:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34510
I20260619 04:36:33.068140 15297 tablet_replica.cc:330] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425: stopping tablet replica
I20260619 04:36:33.069406 15298 tablet_replica.cc:330] T 9c21326af9ff4eb991716ab8c07f8792 P 40bcd4c6d5ff49a795b78b2758d675f5: stopping tablet replica
I20260619 04:36:33.069983 15297 raft_consensus.cc:2229] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:36:33.070577 15298 raft_consensus.cc:2229] T 9c21326af9ff4eb991716ab8c07f8792 P 40bcd4c6d5ff49a795b78b2758d675f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:36:33.072376 15297 raft_consensus.cc:2258] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:36:33.072875 15298 raft_consensus.cc:2258] T 9c21326af9ff4eb991716ab8c07f8792 P 40bcd4c6d5ff49a795b78b2758d675f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:36:33.076210 15298 ts_tablet_manager.cc:1905] T 9c21326af9ff4eb991716ab8c07f8792 P 40bcd4c6d5ff49a795b78b2758d675f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:36:33.079340 15297 ts_tablet_manager.cc:1905] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:36:33.080520 15299 tablet_replica.cc:330] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498: stopping tablet replica
I20260619 04:36:33.080744 15299 raft_consensus.cc:2229] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:36:33.081089 15299 raft_consensus.cc:2258] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:36:33.083804 15299 ts_tablet_manager.cc:1905] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:36:33.085340 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:33.068505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34516 (local address 127.0.97.1:38405)
0619 04:36:33.068876 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:33.068881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:33.068894 (+ 13us) server_negotiation.cc:407] Connection header received
0619 04:36:33.068941 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:33.068944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:33.069001 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:33.069080 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:33.070939 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:33.071658 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:33.073466 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:33.073962 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:33.074088 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:33.077755 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:33.077776 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:33.077792 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:33.077842 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:33.081748 (+ 3906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:33.083021 (+ 1273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:33.083031 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:33.083035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:33.083188 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:33.083605 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:33.083609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:33.083613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:33.084142 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:33.084428 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:33.084808 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 04:36:33.085035 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254}
I20260619 04:36:33.086879 14988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:33.089823 15297 ts_tablet_manager.cc:1918] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:36:33.090034 15297 log.cc:1196] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/9c21326af9ff4eb991716ab8c07f8792
I20260619 04:36:33.090646 15297 ts_tablet_manager.cc:1939] T 9c21326af9ff4eb991716ab8c07f8792 P 6ba2f932b2b84317bee8e5ab65234425: Deleting consensus metadata
I20260619 04:36:33.092043 15298 ts_tablet_manager.cc:1918] T 9c21326af9ff4eb991716ab8c07f8792 P 40bcd4c6d5ff49a795b78b2758d675f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:36:33.092320 15298 log.cc:1196] T 9c21326af9ff4eb991716ab8c07f8792 P 40bcd4c6d5ff49a795b78b2758d675f5: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/9c21326af9ff4eb991716ab8c07f8792
I20260619 04:36:33.092800 14791 catalog_manager.cc:4841] TS 6ba2f932b2b84317bee8e5ab65234425 (127.0.97.3:44013): tablet 9c21326af9ff4eb991716ab8c07f8792 (table test-table [id=c416c502fca046519d83fbd8e34270d7]) successfully deleted
I20260619 04:36:33.093740 15298 ts_tablet_manager.cc:1939] T 9c21326af9ff4eb991716ab8c07f8792 P 40bcd4c6d5ff49a795b78b2758d675f5: Deleting consensus metadata
I20260619 04:36:33.095839 14793 catalog_manager.cc:4841] TS 40bcd4c6d5ff49a795b78b2758d675f5 (127.0.97.2:46457): tablet 9c21326af9ff4eb991716ab8c07f8792 (table test-table [id=c416c502fca046519d83fbd8e34270d7]) successfully deleted
I20260619 04:36:33.186525 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 14843
W20260619 04:36:33.201597 14790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:38405 (error 108)
I20260619 04:36:33.202618 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 14998
I20260619 04:36:33.203955 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:33.203396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38405 (local address 127.0.0.1:34518)
0619 04:36:33.203657 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:33.203785 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260619 04:36:33.204223 14790 catalog_manager.cc:4584] TS 3e25c6ffe3c744a69004e210f17fc498 (127.0.97.1:38405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c21326af9ff4eb991716ab8c07f8792: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38405: connect: Connection refused (error 111)
I20260619 04:36:33.216324 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:33.203482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44958 (local address 127.0.97.2:46457)
0619 04:36:33.203666 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:33.203672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:33.203750 (+ 78us) server_negotiation.cc:407] Connection header received
0619 04:36:33.203918 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:33.203924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:33.204007 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:33.204097 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:33.205540 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:33.206285 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:33.207560 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:33.207971 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:33.208079 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:33.210806 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:33.210821 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:33.210836 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:33.210871 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:33.213632 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:33.214324 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:33.214331 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:33.214334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:33.214460 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:33.214819 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:33.214823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:33.214825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:33.215201 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:33.215488 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:33.215783 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 04:36:33.215992 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 04:36:33.217085 15124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:33.258096 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:33.257525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38405 (local address 127.0.0.1:34534)
0619 04:36:33.257779 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:33.257897 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 04:36:33.259260 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:33.258861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38405 (local address 127.0.0.1:34538)
0619 04:36:33.259021 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:33.259110 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 04:36:33.306170 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 14998
I20260619 04:36:33.319940 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:33.319354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38405 (local address 127.0.0.1:34542)
0619 04:36:33.319612 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:33.319749 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 04:36:33.320674 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 15132
I20260619 04:36:33.321046 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:33.320640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38405 (local address 127.0.0.1:34554)
0619 04:36:33.320810 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:33.320898 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 04:36:33.334179 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:33.321500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39584 (local address 127.0.97.3:44013)
0619 04:36:33.321693 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:33.321698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:33.321712 (+ 14us) server_negotiation.cc:407] Connection header received
0619 04:36:33.321768 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:33.321772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:33.321828 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:33.321911 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:33.323203 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:33.324022 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:33.325346 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:33.325785 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:33.325892 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:33.328325 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:33.328335 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:33.328350 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:33.328387 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:33.331421 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:33.332086 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:33.332093 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:33.332095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:33.332249 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:33.332632 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:33.332635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:33.332637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:33.333000 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:33.333295 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:33.333597 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 04:36:33.333817 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 04:36:33.334976 15255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:33.401850 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:33.401332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38405 (local address 127.0.0.1:34568)
0619 04:36:33.401570 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:33.401686 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 04:36:33.402844 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:33.402484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38405 (local address 127.0.0.1:34576)
0619 04:36:33.402663 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:33.402735 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 04:36:33.456097 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 15132
I20260619 04:36:33.472429 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 14773
I20260619 04:36:33.473635 14836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:33.587374 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:33.586417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38405 (local address 127.0.0.1:34582)
0619 04:36:33.586985 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:33.587103 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435}
I20260619 04:36:33.587754 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 14773
I20260619 04:36:33.605472 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41103
--webserver_interface=127.0.97.62
--webserver_port=36555
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:36:33.923825 15305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:33.924126 15305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:33.924248 15305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:33.931591 15305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:36:33.931715 15305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:33.931785 15305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:33.931864 15305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:36:33.931923 15305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:36:33.941984 15305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:33.943325 15305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:33.944707 15305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:33.955953 15310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:33.955977 15313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:33.956017 15311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:33.958297 15305 server_base.cc:1034] running on GCE node
I20260619 04:36:33.959111 15305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:33.960628 15305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:33.961838 15305 hybrid_clock.cc:648] HybridClock initialized: now 1781843793961784 us; error 47 us; skew 500 ppm
I20260619 04:36:33.962193 15305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:33 dist-test-slave-n3tq krb5kdc[14220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:33.970068 15305 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:36:33.971556 15305 webserver.cc:460] Webserver started at http://127.0.97.62:36555/ using document root <none> and password file <none>
I20260619 04:36:33.972230 15305 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:33.972343 15305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:33.977784 15305 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 04:36:33.980590 15320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:33.981927 15305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:36:33.982089 15305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "d6c6cbf4cf7d4996840f70e43e689043"
format_stamp: "Formatted at 2026-06-19 04:36:28 on dist-test-slave-n3tq"
server_key: "492fe3b4eb8d5b9866b9e0d7c5ab9deb"
server_key_iv: "3c83bf2d423230d62bb83c209efacc9f"
server_key_version: "encryptionkey@0"
I20260619 04:36:33.983062 15305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:34.011924 15305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:34.016261 15305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:34.016566 15305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:34.034047 15305 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41103
I20260619 04:36:34.034066 15371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41103 every 8 connection(s)
I20260619 04:36:34.035533 15305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:36:34.040124 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 15305
I20260619 04:36:34.040938 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:38405
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41103
--webserver_port=45233
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:36:34.042618 15372 sys_catalog.cc:263] Verifying existing consensus state
I20260619 04:36:34.045919 15372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043: Bootstrap starting.
I20260619 04:36:34.073581 15372 log.cc:824] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043: Log is configured to *not* fsync() on all Append() calls
I20260619 04:36:34.090090 15372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 04:36:34.090601 15372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043: Bootstrap complete.
I20260619 04:36:34.099529 15372 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c6cbf4cf7d4996840f70e43e689043" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41103 } }
I20260619 04:36:34.100432 15372 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c6cbf4cf7d4996840f70e43e689043, State: Initialized, Role: FOLLOWER
I20260619 04:36:34.101130 15372 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c6cbf4cf7d4996840f70e43e689043" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41103 } }
I20260619 04:36:34.101352 15372 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:36:34.101470 15372 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:36:34.101620 15372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 2 FOLLOWER]: Advancing to term 3
I20260619 04:36:34.105501 15372 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c6cbf4cf7d4996840f70e43e689043" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41103 } }
I20260619 04:36:34.105979 15372 leader_election.cc:302] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6c6cbf4cf7d4996840f70e43e689043; no voters:
I20260619 04:36:34.106603 15372 leader_election.cc:288] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 04:36:34.106751 15376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 04:36:34.107785 15376 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [term 3 LEADER]: Becoming Leader. State: Replica: d6c6cbf4cf7d4996840f70e43e689043, State: Running, Role: LEADER
I20260619 04:36:34.108332 15376 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c6cbf4cf7d4996840f70e43e689043" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41103 } }
I20260619 04:36:34.108814 15372 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:36:34.110853 15378 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6c6cbf4cf7d4996840f70e43e689043. Latest consensus state: current_term: 3 leader_uuid: "d6c6cbf4cf7d4996840f70e43e689043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c6cbf4cf7d4996840f70e43e689043" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41103 } } }
I20260619 04:36:34.111239 15378 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [sys.catalog]: This master's current role is: LEADER
I20260619 04:36:34.111822 15377 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d6c6cbf4cf7d4996840f70e43e689043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c6cbf4cf7d4996840f70e43e689043" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41103 } } }
I20260619 04:36:34.112067 15377 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c6cbf4cf7d4996840f70e43e689043 [sys.catalog]: This master's current role is: LEADER
I20260619 04:36:34.121300 15384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:36:34.131127 15384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:36:34.132632 15384 catalog_manager.cc:1259] Loaded cluster ID: 8f56ce11127643ef9b1d883a073e62f6
I20260619 04:36:34.132728 15384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 04:36:34.134624 15384 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f51e96d49ad google::LogMessage::Fail() at ??:0
@ 0x7f51e96d788a google::LogMessage::SendToLog() at ??:0
@ 0x7f51e96d44de google::LogMessage::Flush() at ??:0
@ 0x7f51e96d5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f51ff10b512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f51ff108976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f51ff190610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f51ff190439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f51ec320345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f51ec32391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f51ec323799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f51ec2feeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f51ef5d86db start_thread at ??:0
@ 0x7f51e856e61f clone at ??:0
W20260619 04:36:34.422500 15374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:34.422803 15374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:34.422935 15374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:34.430749 15374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:34.430872 15374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:34.431041 15374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:36:34.441103 15374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:38405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45233
--tserver_master_addrs=127.0.97.62:41103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:34.442402 15374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:34.443691 15374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:34.456595 15414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:34.456568 15413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:34.456681 15416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:34.458751 15374 server_base.cc:1034] running on GCE node
I20260619 04:36:34.459344 15374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:34.460311 15374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:34.461503 15374 hybrid_clock.cc:648] HybridClock initialized: now 1781843794461439 us; error 71 us; skew 500 ppm
I20260619 04:36:34.461830 15374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:34 dist-test-slave-n3tq krb5kdc[14220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:34.464885 15374 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:36:34.466315 15374 webserver.cc:460] Webserver started at http://127.0.97.1:45233/ using document root <none> and password file <none>
I20260619 04:36:34.466998 15374 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:34.467110 15374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:34.472322 15374 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:36:34.475217 15423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:34.476485 15374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:36:34.476634 15374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "3e25c6ffe3c744a69004e210f17fc498"
format_stamp: "Formatted at 2026-06-19 04:36:28 on dist-test-slave-n3tq"
server_key: "3121e5906846534d2d048cbf58c58d71"
server_key_iv: "a6ffcc1b0704663a204ee1bdb5fc9ac1"
server_key_version: "encryptionkey@0"
I20260619 04:36:34.477557 15374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:34.530769 15374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:34.534752 15374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:34.535008 15374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:34.536069 15374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:36:34.537868 15430 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260619 04:36:34.543658 15374 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 04:36:34.543792 15374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260619 04:36:34.543912 15374 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 04:36:34.547178 15374 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 04:36:34.547272 15374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260619 04:36:34.547572 15430 tablet_bootstrap.cc:492] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498: Bootstrap starting.
I20260619 04:36:34.555761 15430 log.cc:824] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498: Log is configured to *not* fsync() on all Append() calls
I20260619 04:36:34.570340 15430 tablet_bootstrap.cc:492] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 04:36:34.571096 15430 tablet_bootstrap.cc:492] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498: Bootstrap complete.
I20260619 04:36:34.571614 15430 ts_tablet_manager.cc:1397] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498: Time spent bootstrapping tablet: real 0.024s user 0.017s sys 0.004s
I20260619 04:36:34.579635 15430 raft_consensus.cc:348] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } }
I20260619 04:36:34.580696 15430 raft_consensus.cc:729] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e25c6ffe3c744a69004e210f17fc498, State: Initialized, Role: FOLLOWER
I20260619 04:36:34.581748 15430 consensus_queue.cc:260] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } }
I20260619 04:36:34.583767 15430 ts_tablet_manager.cc:1428] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260619 04:36:34.587132 15374 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38405
I20260619 04:36:34.587164 15538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38405 every 8 connection(s)
I20260619 04:36:34.588321 15374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:36:34.592849 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:34.590970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.1:47417)
0619 04:36:34.591734 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:34.591935 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":324,"threads_started":1}
W20260619 04:36:34.593479 15539 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41103 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41103: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
I20260619 04:36:34.594208 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:34.593914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.1:44675)
0619 04:36:34.594043 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:34.594108 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 04:36:34.595000 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:34.594674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.1:38771)
0619 04:36:34.594823 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:34.594890 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 04:36:34.595216 15539 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 04:36:34.597298 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 15374
I20260619 04:36:34.597870 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:46457
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41103
--webserver_port=34815
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:36:34.911134 15542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:34.911413 15542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:34.911506 15542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:34.918622 15542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:34.918726 15542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:34.918838 15542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:36:34.928242 15542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:46457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34815
--tserver_master_addrs=127.0.97.62:41103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:34.929508 15542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:34.930763 15542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:34.943547 15547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:34.943523 15548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:34.943480 15550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:34.944286 15542 server_base.cc:1034] running on GCE node
I20260619 04:36:34.944937 15542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:34.945891 15542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:34.947090 15542 hybrid_clock.cc:648] HybridClock initialized: now 1781843794947035 us; error 47 us; skew 500 ppm
I20260619 04:36:34.947394 15542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:34 dist-test-slave-n3tq krb5kdc[14220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:34.950399 15542 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:36:34.951802 15542 webserver.cc:460] Webserver started at http://127.0.97.2:34815/ using document root <none> and password file <none>
I20260619 04:36:34.952498 15542 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:34.952617 15542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:34.957823 15542 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 04:36:34.960424 15557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:34.961776 15542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:36:34.961923 15542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "40bcd4c6d5ff49a795b78b2758d675f5"
format_stamp: "Formatted at 2026-06-19 04:36:29 on dist-test-slave-n3tq"
server_key: "5ddcf074483b99678223baeb00e08b37"
server_key_iv: "4ee4b7e5aad6090f90a7b04f0970fc32"
server_key_version: "encryptionkey@0"
I20260619 04:36:34.962920 15542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:34.976408 15542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:34.980129 15542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:34.980379 15542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:34.981318 15542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:36:34.982583 15542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:36:34.982673 15542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:34.982779 15542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:36:34.982851 15542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:35.012254 15542 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46457
I20260619 04:36:35.012310 15669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46457 every 8 connection(s)
I20260619 04:36:35.013682 15542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:36:35.018395 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:35.016677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.2:60435)
0619 04:36:35.017364 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:35.017553 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":251,"threads_started":1}
W20260619 04:36:35.018993 15670 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41103 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41103: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
I20260619 04:36:35.019692 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 15542
I20260619 04:36:35.019784 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:35.019418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.2:54321)
0619 04:36:35.019563 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:35.019643 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 04:36:35.020394 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44013
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41103
--webserver_port=44495
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:36:35.020633 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:35.020268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.2:57583)
0619 04:36:35.020380 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:35.020476 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 04:36:35.020872 15670 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 04:36:35.333775 15673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:35.334033 15673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:35.334147 15673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:35.341449 15673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:35.341544 15673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:35.341691 15673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:36:35.351476 15673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44495
--tserver_master_addrs=127.0.97.62:41103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:35.352769 15673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:35.354007 15673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:35.365991 15678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:35.366014 15679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:35.366219 15681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:35.366954 15673 server_base.cc:1034] running on GCE node
I20260619 04:36:35.367492 15673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:35.368345 15673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:35.369567 15673 hybrid_clock.cc:648] HybridClock initialized: now 1781843795369515 us; error 61 us; skew 500 ppm
I20260619 04:36:35.369890 15673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:35 dist-test-slave-n3tq krb5kdc[14220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:35.372884 15673 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:36:35.374231 15673 webserver.cc:460] Webserver started at http://127.0.97.3:44495/ using document root <none> and password file <none>
I20260619 04:36:35.374871 15673 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:35.374989 15673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:35.380033 15673 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:36:35.382634 15688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:35.383985 15673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 04:36:35.384141 15673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "6ba2f932b2b84317bee8e5ab65234425"
format_stamp: "Formatted at 2026-06-19 04:36:29 on dist-test-slave-n3tq"
server_key: "f1046fe1d656c89a13208e406fd8c839"
server_key_iv: "af90ec6885612a9e56fd06bda43d1b28"
server_key_version: "encryptionkey@0"
I20260619 04:36:35.385109 15673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:35.413868 15673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:35.418123 15673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:35.418339 15673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:35.419234 15673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:36:35.420495 15673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:36:35.420607 15673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:35.420693 15673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:36:35.420745 15673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:35.449882 15673 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44013
I20260619 04:36:35.449951 15800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44013 every 8 connection(s)
I20260619 04:36:35.451203 15673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:36:35.453182 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 15673
I20260619 04:36:35.456010 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:35.454118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.3:47573)
0619 04:36:35.454884 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:35.455120 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":308,"threads_started":1}
W20260619 04:36:35.456642 15801 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41103 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41103: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
I20260619 04:36:35.457406 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:35.457080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.3:36707)
0619 04:36:35.457236 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:35.457303 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 04:36:35.458175 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:35.457877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.3:47933)
0619 04:36:35.457999 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:35.458070 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 04:36:35.458372 15801 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 04:36:35.596410 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:35.595727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.1:52259)
0619 04:36:35.596123 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:35.596207 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":184,"threads_started":1}
I20260619 04:36:36.022217 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:36.021420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.2:32807)
0619 04:36:36.021903 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:36.021995 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":247,"threads_started":1}
I20260619 04:36:36.307407 15807 raft_consensus.cc:482] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:36:36.307683 15807 raft_consensus.cc:504] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } }
I20260619 04:36:36.309258 15807 leader_election.cc:288] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ba2f932b2b84317bee8e5ab65234425 (127.0.97.3:44013), 40bcd4c6d5ff49a795b78b2758d675f5 (127.0.97.2:46457)
Jun 19 04:36:36 dist-test-slave-n3tq krb5kdc[14220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
Jun 19 04:36:36 dist-test-slave-n3tq krb5kdc[14220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 04:36:36.333678 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:36.309867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58799 (local address 127.0.97.3:44013)
0619 04:36:36.311305 (+ 1438us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:36.311319 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:36.311350 (+ 31us) server_negotiation.cc:407] Connection header received
0619 04:36:36.313383 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:36.313392 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:36.313823 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:36.314182 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:36.317158 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:36.319023 (+ 1865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:36.321705 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:36.322680 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:36.322858 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:36.327453 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:36.327462 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:36.327477 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:36.327513 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:36.330270 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:36.331002 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:36.331008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:36.331013 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:36.331166 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:36.331564 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:36.331567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:36.331569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:36.332028 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:36.332194 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:36.332993 (+ 799us) server_negotiation.cc:299] Negotiation successful
0619 04:36:36.333286 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1277,"thread_start_us":1180,"threads_started":1}
I20260619 04:36:36.333709 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:36.309490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46457 (local address 127.0.97.1:34665)
0619 04:36:36.310330 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:36.310350 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:36.311022 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:36.314596 (+ 3574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:36.314608 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:36.315782 (+ 1174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:36.316936 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:36.316965 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:36.319247 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:36.319254 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:36.321571 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:36.321592 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:36.322931 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:36.322938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:36.323257 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:36.324177 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:36.324232 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:36.327497 (+ 3265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:36.330518 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:36.330524 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:36.330528 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:36.330915 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:36.331310 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:36.331315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:36.331323 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:36.331469 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:36.332156 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:36.332160 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:36.332504 (+ 344us) client_negotiation.cc:769] Sending connection context
0619 04:36:36.332749 (+ 245us) client_negotiation.cc:240] Negotiation successful
0619 04:36:36.333248 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":347,"spinlock_wait_cycles":92800,"thread_start_us":205,"threads_started":1}
I20260619 04:36:36.333803 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:36.309490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44013 (local address 127.0.97.1:58799)
0619 04:36:36.310015 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:36.310055 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:36.310693 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:36.314596 (+ 3903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:36.314608 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:36.315779 (+ 1171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:36.316936 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:36.316965 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:36.319247 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:36.319254 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:36.321571 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:36.321592 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:36.322887 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:36.322898 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:36.323260 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:36.324177 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:36.324232 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:36.327182 (+ 2950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:36.330396 (+ 3214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:36.330405 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:36.330420 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:36.330877 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:36.331266 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:36.331270 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:36.331274 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:36.331457 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:36.332150 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:36.332157 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:36.332503 (+ 346us) client_negotiation.cc:769] Sending connection context
0619 04:36:36.332749 (+ 246us) client_negotiation.cc:240] Negotiation successful
0619 04:36:36.333438 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":536,"spinlock_wait_cycles":93568,"thread_start_us":663,"threads_started":1}
I20260619 04:36:36.333699 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:36.309937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34665 (local address 127.0.97.2:46457)
0619 04:36:36.311084 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:36.311100 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:36.311147 (+ 47us) server_negotiation.cc:407] Connection header received
0619 04:36:36.313309 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:36.313322 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:36.313825 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:36.314182 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:36.317158 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:36.319023 (+ 1865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:36.321705 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:36.322680 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:36.322851 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:36.327744 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:36.327751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:36.327763 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:36.327801 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:36.330395 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:36.331041 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:36.331047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:36.331052 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:36.331205 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:36.331575 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:36.331578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:36.331580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:36.332038 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:36.332198 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:36.332988 (+ 790us) server_negotiation.cc:299] Negotiation successful
0619 04:36:36.333343 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":976,"thread_start_us":809,"threads_started":1}
W20260619 04:36:36.337005 15425 leader_election.cc:341] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6ba2f932b2b84317bee8e5ab65234425 (127.0.97.3:44013): Not found: Tablet not found: 9c21326af9ff4eb991716ab8c07f8792
W20260619 04:36:36.337255 15427 leader_election.cc:341] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40bcd4c6d5ff49a795b78b2758d675f5 (127.0.97.2:46457): Not found: Tablet not found: 9c21326af9ff4eb991716ab8c07f8792
I20260619 04:36:36.337410 15427 leader_election.cc:302] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e25c6ffe3c744a69004e210f17fc498; no voters: 40bcd4c6d5ff49a795b78b2758d675f5, 6ba2f932b2b84317bee8e5ab65234425
I20260619 04:36:36.337764 15807 raft_consensus.cc:2726] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 04:36:36.459630 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:36.458856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.3:47585)
0619 04:36:36.459285 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:36.459404 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":205,"threads_started":1}
I20260619 04:36:36.597678 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:36.597279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.1:38021)
0619 04:36:36.597444 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:36.597546 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 04:36:37.023692 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:37.023009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.2:40677)
0619 04:36:37.023455 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:37.023533 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":246,"threads_started":1}
I20260619 04:36:37.461234 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:37.460517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.3:55751)
0619 04:36:37.460952 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:37.461033 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":229,"threads_started":1}
I20260619 04:36:37.599390 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:37.598617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.1:55811)
0619 04:36:37.599119 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:37.599203 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":253,"threads_started":1}
I20260619 04:36:38.025259 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:38.024561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.2:35499)
0619 04:36:38.024995 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:38.025091 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":221,"threads_started":1}
I20260619 04:36:38.462865 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:38.462085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.3:47605)
0619 04:36:38.462599 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:38.462683 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":275,"threads_started":1}
I20260619 04:36:38.474921 15818 raft_consensus.cc:482] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:36:38.475114 15818 raft_consensus.cc:504] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } }
I20260619 04:36:38.476171 15818 leader_election.cc:288] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ba2f932b2b84317bee8e5ab65234425 (127.0.97.3:44013), 40bcd4c6d5ff49a795b78b2758d675f5 (127.0.97.2:46457)
W20260619 04:36:38.476809 15425 leader_election.cc:341] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6ba2f932b2b84317bee8e5ab65234425 (127.0.97.3:44013): Not found: Tablet not found: 9c21326af9ff4eb991716ab8c07f8792
W20260619 04:36:38.476981 15427 leader_election.cc:341] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40bcd4c6d5ff49a795b78b2758d675f5 (127.0.97.2:46457): Not found: Tablet not found: 9c21326af9ff4eb991716ab8c07f8792
I20260619 04:36:38.477077 15427 leader_election.cc:302] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e25c6ffe3c744a69004e210f17fc498; no voters: 40bcd4c6d5ff49a795b78b2758d675f5, 6ba2f932b2b84317bee8e5ab65234425
I20260619 04:36:38.477285 15818 raft_consensus.cc:2726] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 04:36:38.600987 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:38.600166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.1:53561)
0619 04:36:38.600686 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:38.600798 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":235,"threads_started":1}
I20260619 04:36:39.026835 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:39.026109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.2:40731)
0619 04:36:39.026560 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:39.026645 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":215,"threads_started":1}
I20260619 04:36:39.464445 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:39.463727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.3:60439)
0619 04:36:39.464165 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:39.464253 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":218,"threads_started":1}
I20260619 04:36:39.602644 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:39.601916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.1:49349)
0619 04:36:39.602380 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:39.602462 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":210,"threads_started":1}
I20260619 04:36:40.028419 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:40.027711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.2:52383)
0619 04:36:40.028169 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:40.028253 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":245,"threads_started":1}
I20260619 04:36:40.466058 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:40.465345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.3:44271)
0619 04:36:40.465815 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:40.465897 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":254,"threads_started":1}
I20260619 04:36:40.604211 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:40.603495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.1:44555)
0619 04:36:40.603918 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:40.604013 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":215,"threads_started":1}
I20260619 04:36:41.030066 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:41.029309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.2:52441)
0619 04:36:41.029775 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:41.029863 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":250,"threads_started":1}
I20260619 04:36:41.467628 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:41.466841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.3:52069)
0619 04:36:41.467372 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:41.467452 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":298,"threads_started":1}
I20260619 04:36:41.605844 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:41.605112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.1:59827)
0619 04:36:41.605552 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:41.605632 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":243,"threads_started":1}
I20260619 04:36:41.651865 15829 raft_consensus.cc:482] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:36:41.652050 15829 raft_consensus.cc:504] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } }
I20260619 04:36:41.652767 15829 leader_election.cc:288] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ba2f932b2b84317bee8e5ab65234425 (127.0.97.3:44013), 40bcd4c6d5ff49a795b78b2758d675f5 (127.0.97.2:46457)
W20260619 04:36:41.653668 15427 leader_election.cc:341] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40bcd4c6d5ff49a795b78b2758d675f5 (127.0.97.2:46457): Not found: Tablet not found: 9c21326af9ff4eb991716ab8c07f8792
W20260619 04:36:41.653839 15425 leader_election.cc:341] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6ba2f932b2b84317bee8e5ab65234425 (127.0.97.3:44013): Not found: Tablet not found: 9c21326af9ff4eb991716ab8c07f8792
I20260619 04:36:41.653961 15425 leader_election.cc:302] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e25c6ffe3c744a69004e210f17fc498; no voters: 40bcd4c6d5ff49a795b78b2758d675f5, 6ba2f932b2b84317bee8e5ab65234425
I20260619 04:36:41.654207 15829 raft_consensus.cc:2726] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 04:36:42.031824 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:42.031017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.2:50357)
0619 04:36:42.031546 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:42.031631 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":234,"threads_started":1}
I20260619 04:36:42.469252 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:42.468474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.3:57583)
0619 04:36:42.468984 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:42.469075 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":221,"threads_started":1}
I20260619 04:36:42.607652 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:42.606803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.1:50585)
0619 04:36:42.607325 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:42.607442 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":257,"threads_started":1}
I20260619 04:36:43.033439 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:43.032743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.2:38035)
0619 04:36:43.033187 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:43.033270 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":228,"threads_started":1}
I20260619 04:36:43.470839 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:43.470151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.3:41301)
0619 04:36:43.470577 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:43.470658 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":222,"threads_started":1}
I20260619 04:36:43.609351 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:43.608629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.1:40291)
0619 04:36:43.609086 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:43.609167 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":229,"threads_started":1}
I20260619 04:36:44.035260 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:44.034456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.2:57033)
0619 04:36:44.034989 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:44.035098 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":235,"threads_started":1}
I20260619 04:36:44.472419 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:44.471700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.3:57447)
0619 04:36:44.472184 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:44.472265 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":213,"threads_started":1}
I20260619 04:36:44.611302 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:44.610415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.1:50121)
0619 04:36:44.610948 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:44.611078 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":283,"threads_started":1}
I20260619 04:36:45.037073 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:45.036293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.2:38063)
0619 04:36:45.036815 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:45.036894 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":296,"threads_started":1}
I20260619 04:36:45.279798 15840 raft_consensus.cc:482] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:36:45.279997 15840 raft_consensus.cc:504] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } }
I20260619 04:36:45.280754 15840 leader_election.cc:288] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ba2f932b2b84317bee8e5ab65234425 (127.0.97.3:44013), 40bcd4c6d5ff49a795b78b2758d675f5 (127.0.97.2:46457)
W20260619 04:36:45.281814 15425 leader_election.cc:341] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6ba2f932b2b84317bee8e5ab65234425 (127.0.97.3:44013): Not found: Tablet not found: 9c21326af9ff4eb991716ab8c07f8792
W20260619 04:36:45.282029 15427 leader_election.cc:341] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40bcd4c6d5ff49a795b78b2758d675f5 (127.0.97.2:46457): Not found: Tablet not found: 9c21326af9ff4eb991716ab8c07f8792
I20260619 04:36:45.282169 15427 leader_election.cc:302] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e25c6ffe3c744a69004e210f17fc498; no voters: 40bcd4c6d5ff49a795b78b2758d675f5, 6ba2f932b2b84317bee8e5ab65234425
I20260619 04:36:45.282424 15840 raft_consensus.cc:2726] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 04:36:45.474056 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:45.473344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.3:49357)
0619 04:36:45.473783 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:45.473865 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":216,"threads_started":1}
I20260619 04:36:45.613209 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:45.612288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.1:43925)
0619 04:36:45.612924 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:45.613008 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":317,"threads_started":1}
I20260619 04:36:46.038668 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:46.037959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.2:53673)
0619 04:36:46.038398 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:46.038491 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":217,"threads_started":1}
I20260619 04:36:46.475811 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:46.475012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.3:58233)
0619 04:36:46.475547 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:46.475650 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":262,"threads_started":1}
I20260619 04:36:46.614717 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:46.614051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.1:56437)
0619 04:36:46.614472 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:46.614555 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":222,"threads_started":1}
I20260619 04:36:47.040376 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:47.039564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.2:41495)
0619 04:36:47.040072 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:47.040154 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":233,"threads_started":1}
I20260619 04:36:47.348184 15847 raft_consensus.cc:482] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:36:47.348402 15847 raft_consensus.cc:504] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2f932b2b84317bee8e5ab65234425" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44013 } } peers { permanent_uuid: "40bcd4c6d5ff49a795b78b2758d675f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46457 } } peers { permanent_uuid: "3e25c6ffe3c744a69004e210f17fc498" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38405 } }
I20260619 04:36:47.349097 15847 leader_election.cc:288] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ba2f932b2b84317bee8e5ab65234425 (127.0.97.3:44013), 40bcd4c6d5ff49a795b78b2758d675f5 (127.0.97.2:46457)
W20260619 04:36:47.349975 15425 leader_election.cc:341] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6ba2f932b2b84317bee8e5ab65234425 (127.0.97.3:44013): Not found: Tablet not found: 9c21326af9ff4eb991716ab8c07f8792
W20260619 04:36:47.350140 15427 leader_election.cc:341] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40bcd4c6d5ff49a795b78b2758d675f5 (127.0.97.2:46457): Not found: Tablet not found: 9c21326af9ff4eb991716ab8c07f8792
I20260619 04:36:47.350288 15427 leader_election.cc:302] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e25c6ffe3c744a69004e210f17fc498; no voters: 40bcd4c6d5ff49a795b78b2758d675f5, 6ba2f932b2b84317bee8e5ab65234425
I20260619 04:36:47.350533 15847 raft_consensus.cc:2726] T 9c21326af9ff4eb991716ab8c07f8792 P 3e25c6ffe3c744a69004e210f17fc498 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 04:36:47.477417 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:47.476708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.3:39863)
0619 04:36:47.477159 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:47.477245 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":213,"threads_started":1}
I20260619 04:36:47.616389 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:47.615629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.1:41063)
0619 04:36:47.616109 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:47.616195 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":230,"threads_started":1}
I20260619 04:36:48.042019 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:48.041301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.2:59955)
0619 04:36:48.041776 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:48.041859 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":238,"threads_started":1}
I20260619 04:36:48.479064 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:48.478360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.3:44023)
0619 04:36:48.478802 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:48.478883 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":225,"threads_started":1}
I20260619 04:36:48.617974 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:48.617269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.1:40471)
0619 04:36:48.617697 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:48.617790 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":231,"threads_started":1}
I20260619 04:36:49.043753 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:49.042970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.2:47365)
0619 04:36:49.043449 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:49.043537 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":235,"threads_started":1}
I20260619 04:36:49.480914 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:49.480027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.3:58639)
0619 04:36:49.480574 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:49.480682 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":245,"threads_started":1}
I20260619 04:36:49.619930 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:49.619190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.1:48825)
0619 04:36:49.619673 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:49.619760 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":231,"threads_started":1}
I20260619 04:36:50.045567 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:50.044873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.2:40443)
0619 04:36:50.045311 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:50.045402 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":211,"threads_started":1}
I20260619 04:36:50.460088 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 15374
I20260619 04:36:50.474767 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:50.461240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39824 (local address 127.0.97.1:38405)
0619 04:36:50.461877 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:50.461884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:50.461913 (+ 29us) server_negotiation.cc:407] Connection header received
0619 04:36:50.461973 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:50.461980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:50.462222 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:50.462387 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:50.464337 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:50.465380 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:50.466677 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:50.467273 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:50.467372 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:50.469850 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:50.469862 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:50.469879 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:50.469911 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:50.472497 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:50.473094 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:50.473101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:50.473103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:50.473241 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:50.473516 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:50.473518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:50.473520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:50.473850 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:50.474087 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:50.474291 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 04:36:50.474488 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":303,"threads_started":1}
I20260619 04:36:50.475975 15534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:50.482761 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:50.482055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41103 (local address 127.0.97.3:55783)
0619 04:36:50.482494 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:50.482594 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":229,"threads_started":1}
I20260619 04:36:50.566679 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 15374
I20260619 04:36:50.581025 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 15542
I20260619 04:36:50.594269 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:50.581758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53170 (local address 127.0.97.2:46457)
0619 04:36:50.582233 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:50.582238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:50.582254 (+ 16us) server_negotiation.cc:407] Connection header received
0619 04:36:50.582308 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:50.582311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:50.582369 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:50.582462 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:50.583779 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:50.584632 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:50.585845 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:50.586281 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:50.586388 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:50.589011 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:50.589016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:50.589019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:50.589055 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:50.591732 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:50.592369 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:50.592374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:50.592376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:50.592530 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:50.592879 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:50.592884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:50.592888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:50.593159 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:50.593420 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:50.593680 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 04:36:50.593893 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":245,"threads_started":1}
I20260619 04:36:50.594944 15665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:50.680742 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 15542
I20260619 04:36:50.695878 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 15673
I20260619 04:36:50.708930 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:50.696677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54014 (local address 127.0.97.3:44013)
0619 04:36:50.697215 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:50.697221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:50.697237 (+ 16us) server_negotiation.cc:407] Connection header received
0619 04:36:50.697292 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:50.697296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:50.697353 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:50.697446 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:50.698860 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:50.699633 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:50.700923 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:50.701353 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:50.701463 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:50.703875 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:50.703881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:50.703884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:50.703917 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:50.706403 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:50.707040 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:50.707045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:50.707047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:50.707179 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:50.707520 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:50.707524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:50.707526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:50.707787 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:50.708050 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:50.708308 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 04:36:50.708556 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":263,"threads_started":1}
I20260619 04:36:50.709599 15796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:50.793434 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 15673
2026-06-19T04:36:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23966 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:36:50 dist-test-slave-n3tq krb5kdc[15866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:36:50 dist-test-slave-n3tq krb5kdc[15866](info): set up 2 sockets
Jun 19 04:36:50 dist-test-slave-n3tq krb5kdc[15866](info): commencing operation
krb5kdc: starting...
W20260619 04:36:51.796751 388 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:36:51 dist-test-slave-n3tq krb5kdc[15866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:36:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:36:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:36:52.011435 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43433
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:36:52.329274 15879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:52.329578 15879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:52.329694 15879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:52.337005 15879 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 04:36:52.337107 15879 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 04:36:52.337163 15879 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 04:36:52.337234 15879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:36:52.337302 15879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:52.337400 15879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:52.337463 15879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:36:52.337536 15879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:36:52.347514 15879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:52.348842 15879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:52.350162 15879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:52.361424 15884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:52.361438 15887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:52.361483 15885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:52.362357 15879 server_base.cc:1034] running on GCE node
I20260619 04:36:52.363101 15879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:52.364481 15879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:52.365693 15879 hybrid_clock.cc:648] HybridClock initialized: now 1781843812365632 us; error 40 us; skew 500 ppm
I20260619 04:36:52.365998 15879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:52 dist-test-slave-n3tq krb5kdc[15866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:52.375540 15879 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:36:52.377070 15879 webserver.cc:460] Webserver started at http://127.0.97.62:44781/ using document root <none> and password file <none>
I20260619 04:36:52.377790 15879 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:52.377884 15879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:52.378206 15879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:36:52.381245 15879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "2dfec6738f1d4b63b701d10fcc07efd8"
format_stamp: "Formatted at 2026-06-19 04:36:52 on dist-test-slave-n3tq"
server_key: "2afdf9a6fbbcb3fe64c6ada60e092168"
server_key_iv: "ed05cb37290d75868864b605aacd2398"
server_key_version: "encryptionkey@0"
I20260619 04:36:52.381994 15879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "2dfec6738f1d4b63b701d10fcc07efd8"
format_stamp: "Formatted at 2026-06-19 04:36:52 on dist-test-slave-n3tq"
server_key: "2afdf9a6fbbcb3fe64c6ada60e092168"
server_key_iv: "ed05cb37290d75868864b605aacd2398"
server_key_version: "encryptionkey@0"
I20260619 04:36:52.386974 15879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:36:52.390349 15894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:52.391674 15879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:36:52.391825 15879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "2dfec6738f1d4b63b701d10fcc07efd8"
format_stamp: "Formatted at 2026-06-19 04:36:52 on dist-test-slave-n3tq"
server_key: "2afdf9a6fbbcb3fe64c6ada60e092168"
server_key_iv: "ed05cb37290d75868864b605aacd2398"
server_key_version: "encryptionkey@0"
I20260619 04:36:52.392019 15879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:52.421867 15879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:52.426072 15879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:52.426321 15879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:52.443639 15879 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43433
I20260619 04:36:52.443658 15945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43433 every 8 connection(s)
I20260619 04:36:52.445003 15879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:36:52.448716 15946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:36:52.452301 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 15879
I20260619 04:36:52.452515 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:36:52.453133 388 external_mini_cluster.cc:1428] Setting key 00d7d38cd19699d44eec878c24230b42
I20260619 04:36:52.459462 15946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8: Bootstrap starting.
I20260619 04:36:52.463158 15946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8: Neither blocks nor log segments found. Creating new log.
I20260619 04:36:52.464344 15946 log.cc:824] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8: Log is configured to *not* fsync() on all Append() calls
I20260619 04:36:52.467736 15946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8: No bootstrap required, opened a new log
Jun 19 04:36:52 dist-test-slave-n3tq krb5kdc[15866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:36:52.477752 15949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:52.454957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.0.97.62:43433)
0619 04:36:52.455682 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:52.455696 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:52.455735 (+ 39us) server_negotiation.cc:407] Connection header received
0619 04:36:52.457939 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:52.457964 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:52.458406 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:52.458748 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:52.460575 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:52.461902 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:52.463751 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:52.464440 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:52.464629 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:52.469139 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:52.469150 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:52.469169 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:52.469229 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:52.473172 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:52.473823 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:52.473835 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:52.473844 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:52.474056 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:52.474362 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:52.474367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:52.474370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:52.475097 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:52.475378 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:52.475778 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 04:36:52.476279 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":325,"threads_started":1}
I20260619 04:36:52.478121 15946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } }
I20260619 04:36:52.478499 15946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:36:52.478601 15946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dfec6738f1d4b63b701d10fcc07efd8, State: Initialized, Role: FOLLOWER
I20260619 04:36:52.479192 15946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } }
I20260619 04:36:52.479450 15946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:36:52.479576 15946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:36:52.479748 15946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:36:52.482107 15946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } }
I20260619 04:36:52.482601 15946 leader_election.cc:302] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dfec6738f1d4b63b701d10fcc07efd8; no voters:
I20260619 04:36:52.483174 15946 leader_election.cc:288] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:36:52.483314 15951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:36:52.483933 15951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 1 LEADER]: Becoming Leader. State: Replica: 2dfec6738f1d4b63b701d10fcc07efd8, State: Running, Role: LEADER
I20260619 04:36:52.484501 15951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } }
I20260619 04:36:52.485189 15946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:36:52.486999 15952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } } }
I20260619 04:36:52.487864 15953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dfec6738f1d4b63b701d10fcc07efd8. Latest consensus state: current_term: 1 leader_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } } }
I20260619 04:36:52.488145 15952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [sys.catalog]: This master's current role is: LEADER
I20260619 04:36:52.488241 15953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [sys.catalog]: This master's current role is: LEADER
I20260619 04:36:52.489892 15958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:36:52.495131 15958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:36:52.503746 15958 catalog_manager.cc:1347] Generated new cluster ID: 91cdcaea8b9a4f9395e552a490828bbd
I20260619 04:36:52.503863 15958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:36:52.508909 15968 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 04:36:52.535962 15958 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:36:52.538007 15958 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:36:52.546005 15958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8: Generated new TSK 0
I20260619 04:36:52.547155 15958 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 04:36:52.606220 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43433
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:36:52.927850 15972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:52.928146 15972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:52.928273 15972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:52.935710 15972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:52.935827 15972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:52.935947 15972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:36:52.945902 15972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:52.947154 15972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:52.948508 15972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:52.961351 15978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:52.961426 15977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:52.961418 15980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:52.962327 15972 server_base.cc:1034] running on GCE node
I20260619 04:36:52.962919 15972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:52.963889 15972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:52.965098 15972 hybrid_clock.cc:648] HybridClock initialized: now 1781843812965031 us; error 43 us; skew 500 ppm
I20260619 04:36:52.965417 15972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:52 dist-test-slave-n3tq krb5kdc[15866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:52.968554 15972 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:36:52.969986 15972 webserver.cc:460] Webserver started at http://127.0.97.1:33197/ using document root <none> and password file <none>
I20260619 04:36:52.970671 15972 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:52.970786 15972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:52.971078 15972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:36:52.974057 15972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "8b3ffdf1457645f7a3a566928d3d3c2f"
format_stamp: "Formatted at 2026-06-19 04:36:52 on dist-test-slave-n3tq"
server_key: "0560416ef9b44134c8bf5e36ae4bcc1a"
server_key_iv: "dc214dc89012c44732d8bc4f291543eb"
server_key_version: "encryptionkey@0"
I20260619 04:36:52.974812 15972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8b3ffdf1457645f7a3a566928d3d3c2f"
format_stamp: "Formatted at 2026-06-19 04:36:52 on dist-test-slave-n3tq"
server_key: "0560416ef9b44134c8bf5e36ae4bcc1a"
server_key_iv: "dc214dc89012c44732d8bc4f291543eb"
server_key_version: "encryptionkey@0"
I20260619 04:36:52.980108 15972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 04:36:52.983649 15987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:52.985147 15972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 04:36:52.985306 15972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "8b3ffdf1457645f7a3a566928d3d3c2f"
format_stamp: "Formatted at 2026-06-19 04:36:52 on dist-test-slave-n3tq"
server_key: "0560416ef9b44134c8bf5e36ae4bcc1a"
server_key_iv: "dc214dc89012c44732d8bc4f291543eb"
server_key_version: "encryptionkey@0"
I20260619 04:36:52.985530 15972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:53.001435 15972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:53.005436 15972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:53.005699 15972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:53.006636 15972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:36:53.007951 15972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:36:53.008028 15972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:53.008145 15972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:36:53.008198 15972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:53.037817 15972 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39759
I20260619 04:36:53.037861 16099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39759 every 8 connection(s)
I20260619 04:36:53.039044 15972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:36:53.039403 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 15972
I20260619 04:36:53.039608 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:36:53.040401 388 external_mini_cluster.cc:1428] Setting key 2f4a6b44d39e6b1ee295741c8461e630
Jun 19 04:36:53 dist-test-slave-n3tq krb5kdc[15866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:36:53.058627 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:53.041541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51073 (local address 127.0.97.62:43433)
0619 04:36:53.042005 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:53.042011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:53.043051 (+ 1040us) server_negotiation.cc:407] Connection header received
0619 04:36:53.044894 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:53.044899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:53.044984 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:53.045096 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:53.046632 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:53.047577 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:53.049189 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:53.049585 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:53.049693 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:53.053133 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:53.053138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:53.053141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:53.053173 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:53.055515 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:53.056342 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:53.056346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:53.056349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:53.056518 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:53.056928 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:53.056933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:53.056937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:53.057220 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:53.057352 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:53.058044 (+ 692us) server_negotiation.cc:299] Negotiation successful
0619 04:36:53.058275 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":223,"threads_started":1}
I20260619 04:36:53.059590 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:53.042125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.1:51073)
0619 04:36:53.042877 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:53.042926 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:53.044636 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:53.045305 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:53.045313 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:53.045892 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:53.046461 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:53.046479 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:53.047707 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:53.047710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:53.049054 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:53.049066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:53.049796 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:53.049803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:53.050009 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:53.050628 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:53.050649 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:53.052918 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:53.055729 (+ 2811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:53.055740 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:53.055758 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:53.056212 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:53.056619 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:53.056624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:53.056628 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:53.056816 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:53.057343 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:53.057350 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:53.057684 (+ 334us) client_negotiation.cc:769] Sending connection context
0619 04:36:53.057920 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 04:36:53.058451 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":302,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 04:36:53.061180 16100 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43433
I20260619 04:36:53.061452 16100 heartbeater.cc:464] Registering TS with master...
I20260619 04:36:53.062180 16100 heartbeater.cc:511] Master 127.0.97.62:43433 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 04:36:53.064329 15911 ts_manager.cc:194] Registered new tserver with Master: 8b3ffdf1457645f7a3a566928d3d3c2f (127.0.97.1:39759)
I20260619 04:36:53.065801 15911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 04:36:53.087507 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43433
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:36:53.402876 16106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:53.403158 16106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:53.403293 16106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:53.410463 16106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:53.410575 16106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:53.410734 16106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:36:53.421487 16106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:53.422809 16106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:53.424121 16106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:53.437197 16114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:53.437181 16112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:53.437500 16111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:53.437803 16106 server_base.cc:1034] running on GCE node
I20260619 04:36:53.438405 16106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:53.439374 16106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:53.440551 16106 hybrid_clock.cc:648] HybridClock initialized: now 1781843813440495 us; error 46 us; skew 500 ppm
I20260619 04:36:53.440869 16106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:53 dist-test-slave-n3tq krb5kdc[15866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:53.444191 16106 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:36:53.446019 16106 webserver.cc:460] Webserver started at http://127.0.97.2:35527/ using document root <none> and password file <none>
I20260619 04:36:53.446867 16106 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:53.447012 16106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:53.447366 16106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:36:53.451242 16106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "1d09e4b187424dfbafc0bbcd98b2c656"
format_stamp: "Formatted at 2026-06-19 04:36:53 on dist-test-slave-n3tq"
server_key: "ceef8873bba095c963df495a9408ff23"
server_key_iv: "72b3734290901d47f8e5f7da4915fdb0"
server_key_version: "encryptionkey@0"
I20260619 04:36:53.452188 16106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1d09e4b187424dfbafc0bbcd98b2c656"
format_stamp: "Formatted at 2026-06-19 04:36:53 on dist-test-slave-n3tq"
server_key: "ceef8873bba095c963df495a9408ff23"
server_key_iv: "72b3734290901d47f8e5f7da4915fdb0"
server_key_version: "encryptionkey@0"
I20260619 04:36:53.458659 16106 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 04:36:53.463058 16121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:53.464510 16106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:36:53.464663 16106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "1d09e4b187424dfbafc0bbcd98b2c656"
format_stamp: "Formatted at 2026-06-19 04:36:53 on dist-test-slave-n3tq"
server_key: "ceef8873bba095c963df495a9408ff23"
server_key_iv: "72b3734290901d47f8e5f7da4915fdb0"
server_key_version: "encryptionkey@0"
I20260619 04:36:53.464843 16106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:53.480001 16106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:53.483875 16106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:53.484112 16106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:53.485028 16106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:36:53.486421 16106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:36:53.486495 16106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:53.486617 16106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:36:53.486671 16106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:53.515156 16106 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45341
I20260619 04:36:53.515192 16233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45341 every 8 connection(s)
I20260619 04:36:53.516439 16106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:36:53.518700 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 16106
I20260619 04:36:53.518904 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:36:53.519462 388 external_mini_cluster.cc:1428] Setting key e4c5a259918abfe349f56370be22d509
Jun 19 04:36:53 dist-test-slave-n3tq krb5kdc[15866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:36:53.536895 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:53.519036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50489 (local address 127.0.97.62:43433)
0619 04:36:53.519200 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:53.519205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:53.520645 (+ 1440us) server_negotiation.cc:407] Connection header received
0619 04:36:53.522749 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:53.522755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:53.522838 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:53.522940 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:53.524358 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:53.525347 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:53.526814 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:53.527375 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:53.527493 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:53.531083 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:53.531089 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:53.531092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:53.531125 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:53.533805 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:53.534685 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:53.534688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:53.534690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:53.534809 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:53.535234 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:53.535237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:53.535239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:53.535482 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:53.535586 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:53.536420 (+ 834us) server_negotiation.cc:299] Negotiation successful
0619 04:36:53.536637 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 04:36:53.538439 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:53.519657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.2:50489)
0619 04:36:53.520490 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:53.520542 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:53.522474 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:53.523117 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:53.523125 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:53.523627 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:53.524191 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:53.524208 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:53.525475 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:53.525479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:53.526686 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:53.526697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:53.527565 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:53.527570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:53.527795 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:53.528515 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:53.528546 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:53.530847 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:53.533970 (+ 3123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:53.533980 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:53.534000 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:53.534586 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:53.534900 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:53.534904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:53.534907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:53.535147 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:53.535584 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:53.535592 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:53.536045 (+ 453us) client_negotiation.cc:769] Sending connection context
0619 04:36:53.536347 (+ 302us) client_negotiation.cc:240] Negotiation successful
0619 04:36:53.536909 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":331,"threads_started":1}
I20260619 04:36:53.540508 16234 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43433
I20260619 04:36:53.540892 16234 heartbeater.cc:464] Registering TS with master...
I20260619 04:36:53.541841 16234 heartbeater.cc:511] Master 127.0.97.62:43433 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 04:36:53.543553 15911 ts_manager.cc:194] Registered new tserver with Master: 1d09e4b187424dfbafc0bbcd98b2c656 (127.0.97.2:45341)
I20260619 04:36:53.544410 15911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 04:36:53.569679 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43433
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:36:53.884387 16239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:36:53.884721 16239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:36:53.884845 16239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:36:53.892321 16239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:36:53.892477 16239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:36:53.892629 16239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:36:53.902734 16239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:36:53.903985 16239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:36:53.905265 16239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:36:53.917411 16247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:53.917524 16244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:36:53.917555 16245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:36:53.918416 16239 server_base.cc:1034] running on GCE node
I20260619 04:36:53.919021 16239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:36:53.919898 16239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:36:53.921121 16239 hybrid_clock.cc:648] HybridClock initialized: now 1781843813921049 us; error 47 us; skew 500 ppm
I20260619 04:36:53.921401 16239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:36:53 dist-test-slave-n3tq krb5kdc[15866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:36:53.924357 16239 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:36:53.925770 16239 webserver.cc:460] Webserver started at http://127.0.97.3:42531/ using document root <none> and password file <none>
I20260619 04:36:53.926440 16239 fs_manager.cc:359] Metadata directory not provided
I20260619 04:36:53.926549 16239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:36:53.926841 16239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:36:53.929843 16239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "e7197a25c401474dbcff59d51a5d69a5"
format_stamp: "Formatted at 2026-06-19 04:36:53 on dist-test-slave-n3tq"
server_key: "15a687eb8db89a2d703cb42100785b4e"
server_key_iv: "717437a6b1d0ff045e0c89f8b26c28cb"
server_key_version: "encryptionkey@0"
I20260619 04:36:53.930583 16239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e7197a25c401474dbcff59d51a5d69a5"
format_stamp: "Formatted at 2026-06-19 04:36:53 on dist-test-slave-n3tq"
server_key: "15a687eb8db89a2d703cb42100785b4e"
server_key_iv: "717437a6b1d0ff045e0c89f8b26c28cb"
server_key_version: "encryptionkey@0"
I20260619 04:36:53.935696 16239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 04:36:53.939160 16254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:53.940574 16239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:36:53.940721 16239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "e7197a25c401474dbcff59d51a5d69a5"
format_stamp: "Formatted at 2026-06-19 04:36:53 on dist-test-slave-n3tq"
server_key: "15a687eb8db89a2d703cb42100785b4e"
server_key_iv: "717437a6b1d0ff045e0c89f8b26c28cb"
server_key_version: "encryptionkey@0"
I20260619 04:36:53.940918 16239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:36:53.964039 16239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:36:53.968443 16239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:36:53.968735 16239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:36:53.969692 16239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:36:53.971007 16239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:36:53.971078 16239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:53.971203 16239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:36:53.971253 16239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:36:54.000552 16239 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43063
I20260619 04:36:54.000627 16366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43063 every 8 connection(s)
I20260619 04:36:54.001874 16239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:36:54.010329 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 16239
I20260619 04:36:54.010509 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:36:54.010927 388 external_mini_cluster.cc:1428] Setting key 3f8cadc1a792b0075a169e0b2a527164
Jun 19 04:36:54 dist-test-slave-n3tq krb5kdc[15866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:36:54.021034 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:54.004307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38915 (local address 127.0.97.62:43433)
0619 04:36:54.004548 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:54.004554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:54.005737 (+ 1183us) server_negotiation.cc:407] Connection header received
0619 04:36:54.007508 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:54.007513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:54.007584 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:54.007672 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:54.009209 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.009965 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.011682 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.012224 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.012354 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:54.015917 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:54.015922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:54.015925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:54.015960 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:54.018476 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:54.019141 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:54.019144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:54.019146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:54.019251 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:54.019583 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:54.019586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:54.019588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:54.019841 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:54.019946 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:54.020613 (+ 667us) server_negotiation.cc:299] Negotiation successful
0619 04:36:54.020786 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 04:36:54.022022 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:54.004889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.3:38915)
0619 04:36:54.005577 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:54.005619 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:54.007246 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:54.007882 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:54.007890 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:54.008416 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:36:54.009024 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:54.009041 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.010136 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.010140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:54.011533 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:54.011549 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.012418 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.012423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:54.012688 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:54.013358 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:36:54.013386 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:36:54.015717 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:36:54.018614 (+ 2897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:54.018621 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:54.018636 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:54.019049 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:54.019328 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:36:54.019331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:36:54.019333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:36:54.019505 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:36:54.019929 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:36:54.019934 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:36:54.020259 (+ 325us) client_negotiation.cc:769] Sending connection context
0619 04:36:54.020522 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 04:36:54.020933 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":261,"threads_started":1}
I20260619 04:36:54.023407 16367 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43433
I20260619 04:36:54.023675 16367 heartbeater.cc:464] Registering TS with master...
I20260619 04:36:54.024398 16367 heartbeater.cc:511] Master 127.0.97.62:43433 requested a full tablet report, sending...
I20260619 04:36:54.025930 15911 ts_manager.cc:194] Registered new tserver with Master: e7197a25c401474dbcff59d51a5d69a5 (127.0.97.3:43063)
I20260619 04:36:54.026459 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:36:54.026787 15911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38915
I20260619 04:36:54.043419 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:54.031272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.0.97.62:43433)
0619 04:36:54.031471 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:54.031478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:54.031799 (+ 321us) server_negotiation.cc:407] Connection header received
0619 04:36:54.031933 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:54.031937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:54.032009 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:54.032084 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:54.033484 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.034169 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.035478 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.035886 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.035973 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:54.038519 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:54.038524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:54.038527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:54.038555 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:54.041021 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:54.041747 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:54.041750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:54.041752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:54.041867 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:54.042216 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:54.042218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:54.042220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:54.042509 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:54.042677 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:54.043020 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 04:36:54.043179 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:36:54.047266 15911 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:36:54.054102 15911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:36:54.068703 16100 heartbeater.cc:503] Master 127.0.97.62:43433 was elected leader, sending a full tablet report...
I20260619 04:36:54.075316 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:54.065669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45341 (local address 127.0.0.1:37206)
0619 04:36:54.066533 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:54.066580 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:54.066768 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:54.067803 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:54.067810 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:54.067839 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:36:54.068204 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:54.068216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.070160 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.070166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:54.072618 (+ 2452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:54.072633 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.074459 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.074467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:54.074677 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:54.074735 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 04:36:54.074870 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 04:36:54.075022 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":395,"threads_started":1}
I20260619 04:36:54.075453 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:54.065884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37206 (local address 127.0.97.2:45341)
0619 04:36:54.067196 (+ 1312us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:54.067205 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:54.067232 (+ 27us) server_negotiation.cc:407] Connection header received
0619 04:36:54.067324 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:54.067332 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:54.067587 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:54.067790 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:36:54.068447 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.070017 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.072779 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.074237 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.074353 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:54.074558 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:54.074954 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 04:36:54.075131 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1163,"thread_start_us":1035,"threads_started":1}
I20260619 04:36:54.076503 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:54.067357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36454 (local address 127.0.97.1:39759)
0619 04:36:54.067877 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:54.067886 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:54.067910 (+ 24us) server_negotiation.cc:407] Connection header received
0619 04:36:54.067998 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:54.068005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:54.068287 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:54.069065 (+ 778us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:36:54.069372 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.070847 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.073183 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.074514 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.074630 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:54.074849 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:54.076021 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0619 04:36:54.076183 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":1342,"threads_started":1}
I20260619 04:36:54.076526 16381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:54.066925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39759 (local address 127.0.0.1:36454)
0619 04:36:54.067346 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:54.067359 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:54.067438 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:54.068493 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:54.068498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:54.068521 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:36:54.068873 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:54.068884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.070999 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.071004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:54.073037 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:54.073048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.074720 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.074726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:54.075856 (+ 1130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:54.075880 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 04:36:54.075957 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 04:36:54.076085 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":942,"thread_start_us":591,"threads_started":1}
I20260619 04:36:54.079015 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:54.067811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43063 (local address 127.0.0.1:54502)
0619 04:36:54.068400 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:54.068415 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:54.068558 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:54.071750 (+ 3192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:54.071754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:54.071784 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:36:54.072199 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:54.072209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.074331 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.074336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:54.076447 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:54.076482 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.078209 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.078215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:54.078442 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:54.078475 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 04:36:54.078541 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 04:36:54.078663 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":20,"spinlock_wait_cycles":1792,"thread_start_us":2812,"threads_started":1}
I20260619 04:36:54.080001 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:54.068356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54502 (local address 127.0.97.3:43063)
0619 04:36:54.071313 (+ 2957us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:54.071319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:54.071344 (+ 25us) server_negotiation.cc:407] Connection header received
0619 04:36:54.071414 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:54.071419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:54.071580 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:54.071744 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:36:54.072367 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.074168 (+ 1801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.076653 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.077982 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.079248 (+ 1266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:54.079476 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:54.079542 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 04:36:54.079698 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2825,"thread_start_us":2651,"threads_started":1}
I20260619 04:36:54.079663 16169 tablet_service.cc:1467] Processing CreateTablet for tablet 1987945ecc264c4cafb636f2dc138389 (DEFAULT_TABLE table=test-table [id=9ce1ef6075ba4f8abefa7d40fc70e3d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:36:54.080017 16035 tablet_service.cc:1467] Processing CreateTablet for tablet 1987945ecc264c4cafb636f2dc138389 (DEFAULT_TABLE table=test-table [id=9ce1ef6075ba4f8abefa7d40fc70e3d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:36:54.081254 16169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1987945ecc264c4cafb636f2dc138389. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:36:54.081341 16035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1987945ecc264c4cafb636f2dc138389. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:36:54.083894 16302 tablet_service.cc:1467] Processing CreateTablet for tablet 1987945ecc264c4cafb636f2dc138389 (DEFAULT_TABLE table=test-table [id=9ce1ef6075ba4f8abefa7d40fc70e3d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:36:54.085991 16302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1987945ecc264c4cafb636f2dc138389. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:36:54.093032 16385 tablet_bootstrap.cc:492] T 1987945ecc264c4cafb636f2dc138389 P 1d09e4b187424dfbafc0bbcd98b2c656: Bootstrap starting.
I20260619 04:36:54.095533 16386 tablet_bootstrap.cc:492] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f: Bootstrap starting.
I20260619 04:36:54.096639 16385 tablet_bootstrap.cc:654] T 1987945ecc264c4cafb636f2dc138389 P 1d09e4b187424dfbafc0bbcd98b2c656: Neither blocks nor log segments found. Creating new log.
I20260619 04:36:54.097900 16385 log.cc:824] T 1987945ecc264c4cafb636f2dc138389 P 1d09e4b187424dfbafc0bbcd98b2c656: Log is configured to *not* fsync() on all Append() calls
I20260619 04:36:54.099108 16386 tablet_bootstrap.cc:654] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f: Neither blocks nor log segments found. Creating new log.
I20260619 04:36:54.100287 16388 tablet_bootstrap.cc:492] T 1987945ecc264c4cafb636f2dc138389 P e7197a25c401474dbcff59d51a5d69a5: Bootstrap starting.
I20260619 04:36:54.100502 16386 log.cc:824] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f: Log is configured to *not* fsync() on all Append() calls
I20260619 04:36:54.103708 16388 tablet_bootstrap.cc:654] T 1987945ecc264c4cafb636f2dc138389 P e7197a25c401474dbcff59d51a5d69a5: Neither blocks nor log segments found. Creating new log.
I20260619 04:36:54.103732 16385 tablet_bootstrap.cc:492] T 1987945ecc264c4cafb636f2dc138389 P 1d09e4b187424dfbafc0bbcd98b2c656: No bootstrap required, opened a new log
I20260619 04:36:54.104000 16385 ts_tablet_manager.cc:1397] T 1987945ecc264c4cafb636f2dc138389 P 1d09e4b187424dfbafc0bbcd98b2c656: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260619 04:36:54.104543 16386 tablet_bootstrap.cc:492] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f: No bootstrap required, opened a new log
I20260619 04:36:54.104750 16388 log.cc:824] T 1987945ecc264c4cafb636f2dc138389 P e7197a25c401474dbcff59d51a5d69a5: Log is configured to *not* fsync() on all Append() calls
I20260619 04:36:54.104812 16386 ts_tablet_manager.cc:1397] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260619 04:36:54.107070 16388 tablet_bootstrap.cc:492] T 1987945ecc264c4cafb636f2dc138389 P e7197a25c401474dbcff59d51a5d69a5: No bootstrap required, opened a new log
I20260619 04:36:54.107580 16388 ts_tablet_manager.cc:1397] T 1987945ecc264c4cafb636f2dc138389 P e7197a25c401474dbcff59d51a5d69a5: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.004s
I20260619 04:36:54.114440 16385 raft_consensus.cc:348] T 1987945ecc264c4cafb636f2dc138389 P 1d09e4b187424dfbafc0bbcd98b2c656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:36:54.114931 16385 raft_consensus.cc:374] T 1987945ecc264c4cafb636f2dc138389 P 1d09e4b187424dfbafc0bbcd98b2c656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:36:54.115062 16385 raft_consensus.cc:729] T 1987945ecc264c4cafb636f2dc138389 P 1d09e4b187424dfbafc0bbcd98b2c656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d09e4b187424dfbafc0bbcd98b2c656, State: Initialized, Role: FOLLOWER
I20260619 04:36:54.115159 16386 raft_consensus.cc:348] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:36:54.115574 16386 raft_consensus.cc:374] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:36:54.115666 16386 raft_consensus.cc:729] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b3ffdf1457645f7a3a566928d3d3c2f, State: Initialized, Role: FOLLOWER
I20260619 04:36:54.115461 16388 raft_consensus.cc:348] T 1987945ecc264c4cafb636f2dc138389 P e7197a25c401474dbcff59d51a5d69a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:36:54.115847 16388 raft_consensus.cc:374] T 1987945ecc264c4cafb636f2dc138389 P e7197a25c401474dbcff59d51a5d69a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:36:54.115736 16385 consensus_queue.cc:260] T 1987945ecc264c4cafb636f2dc138389 P 1d09e4b187424dfbafc0bbcd98b2c656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:36:54.115935 16388 raft_consensus.cc:729] T 1987945ecc264c4cafb636f2dc138389 P e7197a25c401474dbcff59d51a5d69a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7197a25c401474dbcff59d51a5d69a5, State: Initialized, Role: FOLLOWER
I20260619 04:36:54.116247 16386 consensus_queue.cc:260] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:36:54.116489 16388 consensus_queue.cc:260] T 1987945ecc264c4cafb636f2dc138389 P e7197a25c401474dbcff59d51a5d69a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:36:54.117187 16234 heartbeater.cc:503] Master 127.0.97.62:43433 was elected leader, sending a full tablet report...
I20260619 04:36:54.117905 16385 ts_tablet_manager.cc:1428] T 1987945ecc264c4cafb636f2dc138389 P 1d09e4b187424dfbafc0bbcd98b2c656: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
I20260619 04:36:54.121868 16386 ts_tablet_manager.cc:1428] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260619 04:36:54.122961 16367 heartbeater.cc:503] Master 127.0.97.62:43433 was elected leader, sending a full tablet report...
I20260619 04:36:54.123668 16388 ts_tablet_manager.cc:1428] T 1987945ecc264c4cafb636f2dc138389 P e7197a25c401474dbcff59d51a5d69a5: Time spent starting tablet: real 0.016s user 0.006s sys 0.006s
W20260619 04:36:54.253399 16368 tablet.cc:2369] T 1987945ecc264c4cafb636f2dc138389 P e7197a25c401474dbcff59d51a5d69a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:36:54.268334 16235 tablet.cc:2369] T 1987945ecc264c4cafb636f2dc138389 P 1d09e4b187424dfbafc0bbcd98b2c656: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:36:54.291365 16101 tablet.cc:2369] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:36:54.421357 16392 raft_consensus.cc:482] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:36:54.421646 16392 raft_consensus.cc:504] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:36:54.423267 16392 leader_election.cc:288] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d09e4b187424dfbafc0bbcd98b2c656 (127.0.97.2:45341), e7197a25c401474dbcff59d51a5d69a5 (127.0.97.3:43063)
I20260619 04:36:54.429652 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:54.423793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50557 (local address 127.0.97.2:45341)
0619 04:36:54.424029 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:54.424035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:54.424393 (+ 358us) server_negotiation.cc:407] Connection header received
0619 04:36:54.424655 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:54.424663 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:54.424751 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:54.424844 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:36:54.425418 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.426148 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.427879 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.428774 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.428893 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:54.428986 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:54.429247 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 04:36:54.429393 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 04:36:54.429802 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:54.423660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45341 (local address 127.0.97.1:50557)
0619 04:36:54.424270 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:54.424286 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:54.424483 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:54.424941 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:54.424946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:54.424974 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:36:54.425285 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:54.425294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.426272 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.426278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:54.427740 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:54.427752 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.428966 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.428972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:54.429105 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:54.429128 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 04:36:54.429261 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 04:36:54.429393 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":3,"thread_start_us":326,"threads_started":1}
I20260619 04:36:54.429934 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:54.424700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52193 (local address 127.0.97.3:43063)
0619 04:36:54.424896 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:54.424902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:54.425100 (+ 198us) server_negotiation.cc:407] Connection header received
0619 04:36:54.425266 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:54.425269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:54.425327 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:54.425399 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:36:54.425868 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.426579 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.428228 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.429038 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.429159 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:54.429272 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:54.429587 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 04:36:54.429705 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 04:36:54.430140 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:54.424398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43063 (local address 127.0.97.1:52193)
0619 04:36:54.425010 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0619 04:36:54.425023 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 04:36:54.425148 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:36:54.425429 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:36:54.425433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:36:54.425455 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:36:54.425732 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:54.425740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.426716 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.426720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:54.428051 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:36:54.428062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.429299 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.429307 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:36:54.429435 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:54.429457 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 04:36:54.429537 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 04:36:54.429669 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":1,"thread_start_us":324,"threads_started":1}
I20260619 04:36:54.430755 16189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1987945ecc264c4cafb636f2dc138389" candidate_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" is_pre_election: true
I20260619 04:36:54.430977 16322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1987945ecc264c4cafb636f2dc138389" candidate_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7197a25c401474dbcff59d51a5d69a5" is_pre_election: true
I20260619 04:36:54.431226 16189 raft_consensus.cc:2454] T 1987945ecc264c4cafb636f2dc138389 P 1d09e4b187424dfbafc0bbcd98b2c656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b3ffdf1457645f7a3a566928d3d3c2f in term 0.
I20260619 04:36:54.431427 16322 raft_consensus.cc:2454] T 1987945ecc264c4cafb636f2dc138389 P e7197a25c401474dbcff59d51a5d69a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b3ffdf1457645f7a3a566928d3d3c2f in term 0.
I20260619 04:36:54.431890 15988 leader_election.cc:302] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d09e4b187424dfbafc0bbcd98b2c656, 8b3ffdf1457645f7a3a566928d3d3c2f; no voters:
I20260619 04:36:54.432313 16392 raft_consensus.cc:2781] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:36:54.432489 16392 raft_consensus.cc:482] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:36:54.432597 16392 raft_consensus.cc:3037] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:36:54.434755 16392 raft_consensus.cc:504] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:36:54.435388 16392 leader_election.cc:288] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 1 election: Requested vote from peers 1d09e4b187424dfbafc0bbcd98b2c656 (127.0.97.2:45341), e7197a25c401474dbcff59d51a5d69a5 (127.0.97.3:43063)
I20260619 04:36:54.435822 16189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1987945ecc264c4cafb636f2dc138389" candidate_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d09e4b187424dfbafc0bbcd98b2c656"
I20260619 04:36:54.435886 16322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1987945ecc264c4cafb636f2dc138389" candidate_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7197a25c401474dbcff59d51a5d69a5"
I20260619 04:36:54.435988 16189 raft_consensus.cc:3037] T 1987945ecc264c4cafb636f2dc138389 P 1d09e4b187424dfbafc0bbcd98b2c656 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:36:54.436070 16322 raft_consensus.cc:3037] T 1987945ecc264c4cafb636f2dc138389 P e7197a25c401474dbcff59d51a5d69a5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:36:54.438130 16322 raft_consensus.cc:2454] T 1987945ecc264c4cafb636f2dc138389 P e7197a25c401474dbcff59d51a5d69a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b3ffdf1457645f7a3a566928d3d3c2f in term 1.
I20260619 04:36:54.438141 16189 raft_consensus.cc:2454] T 1987945ecc264c4cafb636f2dc138389 P 1d09e4b187424dfbafc0bbcd98b2c656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b3ffdf1457645f7a3a566928d3d3c2f in term 1.
I20260619 04:36:54.438488 15988 leader_election.cc:302] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d09e4b187424dfbafc0bbcd98b2c656, 8b3ffdf1457645f7a3a566928d3d3c2f; no voters:
I20260619 04:36:54.438773 16392 raft_consensus.cc:2781] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:36:54.439220 16392 raft_consensus.cc:686] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 LEADER]: Becoming Leader. State: Replica: 8b3ffdf1457645f7a3a566928d3d3c2f, State: Running, Role: LEADER
I20260619 04:36:54.439678 16392 consensus_queue.cc:237] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:36:54.444289 15911 catalog_manager.cc:5495] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b3ffdf1457645f7a3a566928d3d3c2f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } health_report { overall_health: UNKNOWN } } }
I20260619 04:36:54.532657 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:54.527680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36468 (local address 127.0.97.1:39759)
0619 04:36:54.527858 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:54.527862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:54.527961 (+ 99us) server_negotiation.cc:407] Connection header received
0619 04:36:54.528162 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:54.528166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:54.528262 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:54.528384 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:36:54.528764 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.529496 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.530891 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.531262 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.531349 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:54.531714 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:36:54.532216 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:36:54.532322 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:54.532364 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 04:36:54.532430 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 04:36:54.540812 16189 raft_consensus.cc:1261] T 1987945ecc264c4cafb636f2dc138389 P 1d09e4b187424dfbafc0bbcd98b2c656 [term 1 FOLLOWER]: Refusing update from remote peer 8b3ffdf1457645f7a3a566928d3d3c2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:36:54.540812 16322 raft_consensus.cc:1261] T 1987945ecc264c4cafb636f2dc138389 P e7197a25c401474dbcff59d51a5d69a5 [term 1 FOLLOWER]: Refusing update from remote peer 8b3ffdf1457645f7a3a566928d3d3c2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:36:54.541692 16392 consensus_queue.cc:1035] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:36:54.541990 16396 consensus_queue.cc:1035] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:36:54.562021 16399 mvcc.cc:205] Tried to move back new op lower bound from 7298432264348819456 to 7298432263948345344. Current Snapshot: MvccSnapshot[applied={T|T < 7298432264348819456}]
I20260619 04:36:54.563059 16401 mvcc.cc:205] Tried to move back new op lower bound from 7298432264348819456 to 7298432263948345344. Current Snapshot: MvccSnapshot[applied={T|T < 7298432264348819456}]
I20260619 04:36:54.565377 16400 mvcc.cc:205] Tried to move back new op lower bound from 7298432264348819456 to 7298432263948345344. Current Snapshot: MvccSnapshot[applied={T|T < 7298432264348819456}]
I20260619 04:36:54.574595 15911 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:36:54.574903 15911 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:36:54.578529 15911 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8: Sending DeleteTablet for 3 replicas of tablet 1987945ecc264c4cafb636f2dc138389
I20260619 04:36:54.579794 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 15972
I20260619 04:36:54.579713 16169 tablet_service.cc:1514] Processing DeleteTablet for tablet 1987945ecc264c4cafb636f2dc138389 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:36:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37206
I20260619 04:36:54.579865 16302 tablet_service.cc:1514] Processing DeleteTablet for tablet 1987945ecc264c4cafb636f2dc138389 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:36:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54502
I20260619 04:36:54.581090 16035 tablet_service.cc:1514] Processing DeleteTablet for tablet 1987945ecc264c4cafb636f2dc138389 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:36:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36454
I20260619 04:36:54.585206 16411 tablet_replica.cc:330] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f: stopping tablet replica
I20260619 04:36:54.585752 16411 raft_consensus.cc:2229] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:36:54.586568 16411 raft_consensus.cc:2258] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:36:54.591873 16411 ts_tablet_manager.cc:1905] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:36:54.595839 16408 tablet_replica.cc:330] T 1987945ecc264c4cafb636f2dc138389 P 1d09e4b187424dfbafc0bbcd98b2c656: stopping tablet replica
I20260619 04:36:54.596067 16409 tablet_replica.cc:330] T 1987945ecc264c4cafb636f2dc138389 P e7197a25c401474dbcff59d51a5d69a5: stopping tablet replica
I20260619 04:36:54.596300 16408 raft_consensus.cc:2229] T 1987945ecc264c4cafb636f2dc138389 P 1d09e4b187424dfbafc0bbcd98b2c656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:36:54.596510 16409 raft_consensus.cc:2229] T 1987945ecc264c4cafb636f2dc138389 P e7197a25c401474dbcff59d51a5d69a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:36:54.596930 16408 raft_consensus.cc:2258] T 1987945ecc264c4cafb636f2dc138389 P 1d09e4b187424dfbafc0bbcd98b2c656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:36:54.597002 16409 raft_consensus.cc:2258] T 1987945ecc264c4cafb636f2dc138389 P e7197a25c401474dbcff59d51a5d69a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:36:54.599565 16409 ts_tablet_manager.cc:1905] T 1987945ecc264c4cafb636f2dc138389 P e7197a25c401474dbcff59d51a5d69a5: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 04:36:54 dist-test-slave-n3tq krb5kdc[15866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 04:36:54.603188 16411 ts_tablet_manager.cc:1918] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:36:54.603418 16411 log.cc:1196] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/wal/wals/1987945ecc264c4cafb636f2dc138389
I20260619 04:36:54.603963 16411 ts_tablet_manager.cc:1939] T 1987945ecc264c4cafb636f2dc138389 P 8b3ffdf1457645f7a3a566928d3d3c2f: Deleting consensus metadata
I20260619 04:36:54.605882 15895 catalog_manager.cc:4841] TS 8b3ffdf1457645f7a3a566928d3d3c2f (127.0.97.1:39759): tablet 1987945ecc264c4cafb636f2dc138389 (table test-table [id=9ce1ef6075ba4f8abefa7d40fc70e3d5]) successfully deleted
I20260619 04:36:54.607687 16408 ts_tablet_manager.cc:1905] T 1987945ecc264c4cafb636f2dc138389 P 1d09e4b187424dfbafc0bbcd98b2c656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:36:54.612049 16409 ts_tablet_manager.cc:1918] T 1987945ecc264c4cafb636f2dc138389 P e7197a25c401474dbcff59d51a5d69a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:36:54.612294 16409 log.cc:1196] T 1987945ecc264c4cafb636f2dc138389 P e7197a25c401474dbcff59d51a5d69a5: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/1987945ecc264c4cafb636f2dc138389
I20260619 04:36:54.612833 16409 ts_tablet_manager.cc:1939] T 1987945ecc264c4cafb636f2dc138389 P e7197a25c401474dbcff59d51a5d69a5: Deleting consensus metadata
I20260619 04:36:54.614122 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:54.581735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36480 (local address 127.0.97.1:39759)
0619 04:36:54.581933 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:54.581939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:54.582151 (+ 212us) server_negotiation.cc:407] Connection header received
0619 04:36:54.582444 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:54.582450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:54.582543 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:54.582615 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:54.586792 (+ 4177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.588096 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.595505 (+ 7409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.596158 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.596315 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:54.601858 (+ 5543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:54.601880 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:54.601901 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:54.601956 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:54.606104 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:54.607126 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:54.607136 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:54.607140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:54.607337 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:54.607903 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:54.607909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:54.607912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:54.608364 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:54.608873 (+ 509us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:54.609613 (+ 740us) server_negotiation.cc:299] Negotiation successful
0619 04:36:54.613752 (+ 4139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 04:36:54.614784 15895 catalog_manager.cc:4841] TS e7197a25c401474dbcff59d51a5d69a5 (127.0.97.3:43063): tablet 1987945ecc264c4cafb636f2dc138389 (table test-table [id=9ce1ef6075ba4f8abefa7d40fc70e3d5]) successfully deleted
I20260619 04:36:54.615116 16408 ts_tablet_manager.cc:1918] T 1987945ecc264c4cafb636f2dc138389 P 1d09e4b187424dfbafc0bbcd98b2c656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:36:54.615267 16408 log.cc:1196] T 1987945ecc264c4cafb636f2dc138389 P 1d09e4b187424dfbafc0bbcd98b2c656: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/1987945ecc264c4cafb636f2dc138389
I20260619 04:36:54.615752 16408 ts_tablet_manager.cc:1939] T 1987945ecc264c4cafb636f2dc138389 P 1d09e4b187424dfbafc0bbcd98b2c656: Deleting consensus metadata
I20260619 04:36:54.615957 16095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:54.617333 15895 catalog_manager.cc:4841] TS 1d09e4b187424dfbafc0bbcd98b2c656 (127.0.97.2:45341): tablet 1987945ecc264c4cafb636f2dc138389 (table test-table [id=9ce1ef6075ba4f8abefa7d40fc70e3d5]) successfully deleted
I20260619 04:36:54.712493 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 15972
I20260619 04:36:54.727875 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 16106
Jun 19 04:36:54 dist-test-slave-n3tq krb5kdc[15866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 04:36:54.741928 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:54.728582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37214 (local address 127.0.97.2:45341)
0619 04:36:54.728752 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:54.728757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:54.728776 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:36:54.728950 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:54.728956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:54.729028 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:54.729123 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:54.730363 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.731059 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.732424 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.732867 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.732972 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:54.736961 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:54.736975 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:54.736991 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:54.737023 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:54.739655 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:54.740252 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:54.740259 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:54.740262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:54.740378 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:54.740729 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:54.740732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:54.740734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:54.741063 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:54.741315 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:54.741514 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 04:36:54.741667 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 04:36:54.742553 16229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:54.830111 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 16106
I20260619 04:36:54.843811 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 16239
Jun 19 04:36:54 dist-test-slave-n3tq krb5kdc[15866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 04:36:54.857648 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:36:54.844660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54518 (local address 127.0.97.3:43063)
0619 04:36:54.844851 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 04:36:54.844858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:36:54.844879 (+ 21us) server_negotiation.cc:407] Connection header received
0619 04:36:54.844975 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:36:54.844980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:36:54.845045 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:36:54.845120 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:36:54.846358 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.847067 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.848281 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:36:54.848705 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:36:54.848795 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:36:54.852820 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:36:54.852835 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:36:54.852852 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:36:54.852891 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:36:54.855342 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:54.855931 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:54.855938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:54.855940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:54.856072 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:36:54.856366 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:36:54.856369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:36:54.856371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:36:54.856735 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:36:54.856936 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 04:36:54.857176 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 04:36:54.857343 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 04:36:54.858283 16362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:54.947463 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 16239
I20260619 04:36:54.960752 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 15879
I20260619 04:36:54.961923 15941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:36:55.072634 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 15879
I20260619 04:37:15.091144 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43433
--webserver_interface=127.0.97.62
--webserver_port=44781
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:37:15.415414 16416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:37:15.415692 16416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:37:15.415828 16416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:37:15.422945 16416 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 04:37:15.423038 16416 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 04:37:15.423097 16416 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 04:37:15.423166 16416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:37:15.423231 16416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:37:15.423327 16416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:37:15.423385 16416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:37:15.423457 16416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:37:15.433251 16416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:37:15.434445 16416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:37:15.435676 16416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:37:15.456187 16422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:15.456223 16423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:15.456249 16425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:37:15.457114 16416 server_base.cc:1034] running on GCE node
I20260619 04:37:15.457969 16416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:37:15.459522 16416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:37:15.460733 16416 hybrid_clock.cc:648] HybridClock initialized: now 1781843835460657 us; error 50 us; skew 500 ppm
I20260619 04:37:15.461094 16416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:37:15 dist-test-slave-n3tq krb5kdc[15866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:37:15.468745 16416 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:37:15.470330 16416 webserver.cc:460] Webserver started at http://127.0.97.62:44781/ using document root <none> and password file <none>
I20260619 04:37:15.471100 16416 fs_manager.cc:359] Metadata directory not provided
I20260619 04:37:15.471208 16416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:37:15.477124 16416 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 04:37:15.479993 16432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:15.481412 16416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:37:15.481572 16416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "2dfec6738f1d4b63b701d10fcc07efd8"
format_stamp: "Formatted at 2026-06-19 04:36:52 on dist-test-slave-n3tq"
server_key: "2afdf9a6fbbcb3fe64c6ada60e092168"
server_key_iv: "ed05cb37290d75868864b605aacd2398"
server_key_version: "encryptionkey@0"
I20260619 04:37:15.482661 16416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:37:15.495877 16416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:37:15.500023 16416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:37:15.500264 16416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:37:15.518164 16416 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43433
I20260619 04:37:15.518198 16483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43433 every 8 connection(s)
I20260619 04:37:15.519724 16416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:37:15.522841 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 16416
I20260619 04:37:15.523603 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39759
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:43433
--webserver_port=33197
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:37:15.526937 16484 sys_catalog.cc:263] Verifying existing consensus state
I20260619 04:37:15.530879 16484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8: Bootstrap starting.
I20260619 04:37:15.550274 16484 log.cc:824] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8: Log is configured to *not* fsync() on all Append() calls
I20260619 04:37:15.562076 16484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 04:37:15.562587 16484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8: Bootstrap complete.
I20260619 04:37:15.571245 16484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } }
I20260619 04:37:15.572088 16484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dfec6738f1d4b63b701d10fcc07efd8, State: Initialized, Role: FOLLOWER
I20260619 04:37:15.572692 16484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } }
I20260619 04:37:15.572917 16484 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:37:15.573024 16484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:37:15.573181 16484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 1 FOLLOWER]: Advancing to term 2
I20260619 04:37:15.576947 16484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } }
I20260619 04:37:15.577432 16484 leader_election.cc:302] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dfec6738f1d4b63b701d10fcc07efd8; no voters:
I20260619 04:37:15.578002 16484 leader_election.cc:288] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 04:37:15.578128 16488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 04:37:15.579238 16488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 2 LEADER]: Becoming Leader. State: Replica: 2dfec6738f1d4b63b701d10fcc07efd8, State: Running, Role: LEADER
I20260619 04:37:15.579846 16488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } }
I20260619 04:37:15.580140 16484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:37:15.582347 16490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dfec6738f1d4b63b701d10fcc07efd8. Latest consensus state: current_term: 2 leader_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } } }
I20260619 04:37:15.582688 16490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [sys.catalog]: This master's current role is: LEADER
I20260619 04:37:15.582383 16489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } } }
I20260619 04:37:15.583655 16489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [sys.catalog]: This master's current role is: LEADER
I20260619 04:37:15.584697 16495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:37:15.595616 16495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:37:15.597340 16495 catalog_manager.cc:1259] Loaded cluster ID: 91cdcaea8b9a4f9395e552a490828bbd
I20260619 04:37:15.597442 16495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:37:15.601533 16495 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:37:15.603236 16495 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8: Loaded TSK: 0
I20260619 04:37:15.629428 16505 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 04:37:15.636888 16495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8: Generated new TSK 1
I20260619 04:37:15.637238 16495 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 04:37:15.876909 16486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:37:15.877198 16486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:37:15.877333 16486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:37:15.884707 16486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:37:15.884826 16486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:37:15.884974 16486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:37:15.895145 16486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39759
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=33197
--tserver_master_addrs=127.0.97.62:43433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:37:15.896479 16486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:37:15.897840 16486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:37:15.912196 16514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:15.912222 16512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:15.912971 16511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:37:15.913582 16486 server_base.cc:1034] running on GCE node
I20260619 04:37:15.914215 16486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:37:15.915184 16486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:37:15.916433 16486 hybrid_clock.cc:648] HybridClock initialized: now 1781843835916365 us; error 59 us; skew 500 ppm
I20260619 04:37:15.916802 16486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:37:15 dist-test-slave-n3tq krb5kdc[15866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:37:15.919904 16486 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:37:15.921499 16486 webserver.cc:460] Webserver started at http://127.0.97.1:33197/ using document root <none> and password file <none>
I20260619 04:37:15.922191 16486 fs_manager.cc:359] Metadata directory not provided
I20260619 04:37:15.922323 16486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:37:15.927671 16486 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 04:37:15.930480 16521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:15.931823 16486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:37:15.931965 16486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "8b3ffdf1457645f7a3a566928d3d3c2f"
format_stamp: "Formatted at 2026-06-19 04:36:52 on dist-test-slave-n3tq"
server_key: "0560416ef9b44134c8bf5e36ae4bcc1a"
server_key_iv: "dc214dc89012c44732d8bc4f291543eb"
server_key_version: "encryptionkey@0"
I20260619 04:37:15.932958 16486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:37:15.952128 16486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:37:15.956248 16486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:37:15.956513 16486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:37:15.957551 16486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:37:15.958920 16486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:37:15.959010 16486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:15.959110 16486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:37:15.959180 16486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:15.989367 16486 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39759
I20260619 04:37:15.989425 16633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39759 every 8 connection(s)
I20260619 04:37:15.990731 16486 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:37:15.995741 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 16486
I20260619 04:37:15.996335 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45341
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:43433
--webserver_port=35527
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:37:16 dist-test-slave-n3tq krb5kdc[15866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:37:16.013871 16637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:15.993631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.1:60337)
0619 04:37:15.994430 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:15.994473 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 04:37:15.996165 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:37:15.997797 (+ 1632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:37:15.997807 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:37:15.998325 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:37:15.998845 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:15.998862 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.000915 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.000919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:16.002140 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:16.002151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.003430 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.003438 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:16.003713 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:16.004350 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:37:16.004377 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:37:16.006493 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:37:16.009710 (+ 3217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:37:16.009719 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:37:16.009735 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:37:16.010135 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:37:16.010549 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:37:16.010554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:37:16.010558 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:37:16.010734 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:37:16.011446 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:37:16.011454 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:37:16.011771 (+ 317us) client_negotiation.cc:769] Sending connection context
0619 04:37:16.012014 (+ 243us) client_negotiation.cc:240] Negotiation successful
0619 04:37:16.012589 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":329,"threads_started":1}
I20260619 04:37:16.013881 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:15.993384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60337 (local address 127.0.97.62:43433)
0619 04:37:15.993923 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:15.993934 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:15.994578 (+ 644us) server_negotiation.cc:407] Connection header received
0619 04:37:15.996766 (+ 2188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:15.996784 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:15.997255 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:15.997639 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:37:15.999022 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.000764 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.002267 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.003189 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.003367 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:16.006752 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:37:16.006760 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:37:16.006775 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:37:16.006821 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:37:16.009573 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:16.010262 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:16.010268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:16.010274 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:16.010434 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:16.010841 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:16.010845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:16.010847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:16.011310 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:37:16.011479 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:16.012217 (+ 738us) server_negotiation.cc:299] Negotiation successful
0619 04:37:16.012613 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":228,"threads_started":1}
I20260619 04:37:16.017086 16634 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43433
I20260619 04:37:16.017365 16634 heartbeater.cc:464] Registering TS with master...
I20260619 04:37:16.018082 16634 heartbeater.cc:511] Master 127.0.97.62:43433 requested a full tablet report, sending...
I20260619 04:37:16.020325 16449 ts_manager.cc:194] Registered new tserver with Master: 8b3ffdf1457645f7a3a566928d3d3c2f (127.0.97.1:39759)
I20260619 04:37:16.022486 16449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60337
W20260619 04:37:16.325773 16638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:37:16.326148 16638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:37:16.326296 16638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:37:16.335319 16638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:37:16.335441 16638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:37:16.335587 16638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:37:16.345389 16638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35527
--tserver_master_addrs=127.0.97.62:43433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:37:16.346642 16638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:37:16.347958 16638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:37:16.359984 16643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:16.359964 16644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:16.360139 16646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:37:16.360759 16638 server_base.cc:1034] running on GCE node
I20260619 04:37:16.361311 16638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:37:16.362167 16638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:37:16.363355 16638 hybrid_clock.cc:648] HybridClock initialized: now 1781843836363306 us; error 38 us; skew 500 ppm
I20260619 04:37:16.363642 16638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:37:16 dist-test-slave-n3tq krb5kdc[15866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:37:16.366482 16638 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:37:16.367861 16638 webserver.cc:460] Webserver started at http://127.0.97.2:35527/ using document root <none> and password file <none>
I20260619 04:37:16.368520 16638 fs_manager.cc:359] Metadata directory not provided
I20260619 04:37:16.368649 16638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:37:16.373735 16638 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 04:37:16.376428 16653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:16.377763 16638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:37:16.377910 16638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "1d09e4b187424dfbafc0bbcd98b2c656"
format_stamp: "Formatted at 2026-06-19 04:36:53 on dist-test-slave-n3tq"
server_key: "ceef8873bba095c963df495a9408ff23"
server_key_iv: "72b3734290901d47f8e5f7da4915fdb0"
server_key_version: "encryptionkey@0"
I20260619 04:37:16.378850 16638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:37:16.389902 16638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:37:16.393657 16638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:37:16.393913 16638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:37:16.394894 16638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:37:16.396311 16638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:37:16.396420 16638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:16.396548 16638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:37:16.396620 16638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:16.426002 16638 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45341
I20260619 04:37:16.426056 16765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45341 every 8 connection(s)
I20260619 04:37:16.427412 16638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:37:16.428934 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 16638
I20260619 04:37:16.429543 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43063
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:43433
--webserver_port=42531
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:37:16 dist-test-slave-n3tq krb5kdc[15866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:37:16.447950 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:16.430042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40411 (local address 127.0.97.62:43433)
0619 04:37:16.430265 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:16.430271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:16.431510 (+ 1239us) server_negotiation.cc:407] Connection header received
0619 04:37:16.433789 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:16.433794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:16.433886 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:16.433972 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:37:16.435849 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.436672 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.438481 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.438971 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.439065 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:16.442720 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:37:16.442725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:37:16.442728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:37:16.442760 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:37:16.445109 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:16.445783 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:16.445787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:16.445790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:16.445913 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:16.446297 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:16.446302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:16.446305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:16.446574 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:37:16.446705 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:16.447370 (+ 665us) server_negotiation.cc:299] Negotiation successful
0619 04:37:16.447600 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 04:37:16.448952 16768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:16.430558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.2:40411)
0619 04:37:16.431325 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:16.431374 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 04:37:16.433501 (+ 2127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:37:16.434232 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:37:16.434244 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:37:16.434944 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:37:16.435679 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:16.435704 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.436816 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.436821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:16.438341 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:16.438353 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.439173 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.439180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:16.439432 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:16.440106 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:37:16.440131 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:37:16.442506 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:37:16.445231 (+ 2725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:37:16.445239 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:37:16.445256 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:37:16.445672 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:37:16.446013 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:37:16.446018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:37:16.446022 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:37:16.446197 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:37:16.446701 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:37:16.446708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:37:16.447029 (+ 321us) client_negotiation.cc:769] Sending connection context
0619 04:37:16.447267 (+ 238us) client_negotiation.cc:240] Negotiation successful
0619 04:37:16.447798 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":295,"threads_started":1}
I20260619 04:37:16.450397 16766 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43433
I20260619 04:37:16.450668 16766 heartbeater.cc:464] Registering TS with master...
I20260619 04:37:16.451334 16766 heartbeater.cc:511] Master 127.0.97.62:43433 requested a full tablet report, sending...
I20260619 04:37:16.452819 16449 ts_manager.cc:194] Registered new tserver with Master: 1d09e4b187424dfbafc0bbcd98b2c656 (127.0.97.2:45341)
I20260619 04:37:16.453938 16449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40411
W20260619 04:37:16.753535 16769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:37:16.753805 16769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:37:16.753896 16769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:37:16.761142 16769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:37:16.761242 16769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:37:16.761359 16769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:37:16.770990 16769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=42531
--tserver_master_addrs=127.0.97.62:43433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:37:16.772156 16769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:37:16.773368 16769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:37:16.785444 16774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:16.785390 16775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:16.785832 16777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:37:16.786439 16769 server_base.cc:1034] running on GCE node
I20260619 04:37:16.787048 16769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:37:16.787928 16769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:37:16.789109 16769 hybrid_clock.cc:648] HybridClock initialized: now 1781843836789062 us; error 41 us; skew 500 ppm
I20260619 04:37:16.789383 16769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:37:16 dist-test-slave-n3tq krb5kdc[15866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:37:16.792250 16769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:37:16.793666 16769 webserver.cc:460] Webserver started at http://127.0.97.3:42531/ using document root <none> and password file <none>
I20260619 04:37:16.794299 16769 fs_manager.cc:359] Metadata directory not provided
I20260619 04:37:16.794420 16769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:37:16.799309 16769 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 04:37:16.801787 16784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:16.802989 16769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 04:37:16.803128 16769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "e7197a25c401474dbcff59d51a5d69a5"
format_stamp: "Formatted at 2026-06-19 04:36:53 on dist-test-slave-n3tq"
server_key: "15a687eb8db89a2d703cb42100785b4e"
server_key_iv: "717437a6b1d0ff045e0c89f8b26c28cb"
server_key_version: "encryptionkey@0"
I20260619 04:37:16.804025 16769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:37:16.828876 16769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:37:16.833062 16769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:37:16.833314 16769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:37:16.834286 16769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:37:16.835587 16769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:37:16.835692 16769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:16.835795 16769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:37:16.835865 16769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:16.865356 16769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43063
I20260619 04:37:16.865420 16896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43063 every 8 connection(s)
I20260619 04:37:16.866763 16769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:37:16.876777 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 16769
Jun 19 04:37:16 dist-test-slave-n3tq krb5kdc[15866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:37:16.887797 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:16.869284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35989 (local address 127.0.97.62:43433)
0619 04:37:16.869465 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:16.869469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:16.870605 (+ 1136us) server_negotiation.cc:407] Connection header received
0619 04:37:16.872437 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:16.872443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:16.872551 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:16.872637 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:37:16.874118 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.874835 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.876383 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.876858 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.876986 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:16.881419 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:37:16.881425 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:37:16.881428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:37:16.881467 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:37:16.884417 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:16.885304 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:16.885310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:16.885313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:16.885475 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:16.885982 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:16.885987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:16.885991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:16.886264 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:37:16.886408 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:16.887238 (+ 830us) server_negotiation.cc:299] Negotiation successful
0619 04:37:16.887441 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:37:16.889055 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:16.869719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.3:35989)
0619 04:37:16.870441 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:16.870484 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 04:37:16.872161 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:37:16.872878 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:37:16.872886 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:37:16.873398 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:37:16.873960 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:16.873977 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.874980 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.874984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:16.876230 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:16.876242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.877081 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.877088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:16.877365 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:16.878155 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:37:16.878187 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:37:16.881227 (+ 3040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:37:16.884555 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:37:16.884564 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:37:16.884584 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:37:16.885211 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:37:16.885619 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:37:16.885624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:37:16.885627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:37:16.885868 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:37:16.886430 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:37:16.886439 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:37:16.886859 (+ 420us) client_negotiation.cc:769] Sending connection context
0619 04:37:16.887158 (+ 299us) client_negotiation.cc:240] Negotiation successful
0619 04:37:16.887686 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":288,"threads_started":1}
I20260619 04:37:16.890573 16897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43433
I20260619 04:37:16.890902 16897 heartbeater.cc:464] Registering TS with master...
I20260619 04:37:16.891758 16897 heartbeater.cc:511] Master 127.0.97.62:43433 requested a full tablet report, sending...
I20260619 04:37:16.893469 16449 ts_manager.cc:194] Registered new tserver with Master: e7197a25c401474dbcff59d51a5d69a5 (127.0.97.3:43063)
I20260619 04:37:16.894522 16449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35989
I20260619 04:37:16.894972 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:16.877645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36830 (local address 127.0.97.62:43433)
0619 04:37:16.878381 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:16.878385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:16.880096 (+ 1711us) server_negotiation.cc:407] Connection header received
0619 04:37:16.880227 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:16.880232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:16.880312 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:16.880411 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:37:16.882349 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.883530 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.885008 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.885619 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.885727 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:16.889402 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:37:16.889410 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:37:16.889415 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:37:16.889449 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:37:16.892391 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:16.893166 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:16.893170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:16.893172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:16.893279 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:16.893687 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:16.893690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:16.893693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:16.893970 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:37:16.894178 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:16.894486 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 04:37:16.894667 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":515,"threads_started":1}
I20260619 04:37:16.897382 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:37:16.903407 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:16.898208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36840 (local address 127.0.97.62:43433)
0619 04:37:16.898356 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:16.898360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:16.898885 (+ 525us) server_negotiation.cc:407] Connection header received
0619 04:37:16.899046 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:16.899049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:16.899169 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:16.899280 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:37:16.899716 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.900389 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.901824 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.902230 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.902328 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:16.902736 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:37:16.902936 (+ 200us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 04:37:16.903192 (+ 256us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36840: token expired
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 04:37:16.903525 16900 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36840: token expired
W20260619 04:37:16.904132 16371 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:43433)
I20260619 04:37:16.916265 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:16.904593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.0.97.62:43433)
0619 04:37:16.904745 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:16.904750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:16.904770 (+ 20us) server_negotiation.cc:407] Connection header received
0619 04:37:16.904920 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:16.904925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:16.904981 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:16.905056 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:37:16.906338 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.907185 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.908515 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.908906 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.909020 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:16.911484 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:37:16.911489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:37:16.911493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:37:16.911524 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:37:16.913805 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:16.914442 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:16.914446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:16.914449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:16.914579 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:16.914904 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:16.914907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:16.914909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:16.915220 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:37:16.915447 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:16.915744 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 04:37:16.915948 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 04:37:16.919564 16449 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:37:16.926681 16449 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:37:16.950100 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:16.938050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39759 (local address 127.0.0.1:49620)
0619 04:37:16.938808 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:16.938846 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 04:37:16.939020 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:37:16.940015 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:37:16.940022 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:37:16.940053 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:37:16.940489 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:16.940503 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.943942 (+ 3439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.943949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:16.946367 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:16.946381 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.949076 (+ 2695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.949082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:16.949295 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:16.949355 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 04:37:16.949501 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 04:37:16.949694 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":388,"threads_started":1}
I20260619 04:37:16.950314 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:16.938163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.0.97.1:39759)
0619 04:37:16.939380 (+ 1217us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:16.939389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:16.939416 (+ 27us) server_negotiation.cc:407] Connection header received
0619 04:37:16.939521 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:16.939530 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:16.939786 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:16.940727 (+ 941us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:37:16.940971 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.943761 (+ 2790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.947444 (+ 3683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.948852 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.948974 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:16.949167 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:16.949540 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 04:37:16.949989 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1070,"thread_start_us":963,"threads_started":1}
I20260619 04:37:16.953399 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:16.940323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44244 (local address 127.0.97.3:43063)
0619 04:37:16.945737 (+ 5414us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:16.945747 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:16.946076 (+ 329us) server_negotiation.cc:407] Connection header received
0619 04:37:16.946171 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:16.946178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:16.946449 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:16.946635 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:37:16.947277 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.948646 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.951101 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.952347 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.952497 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:16.952706 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:16.952945 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 04:37:16.953106 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5223,"thread_start_us":5051,"threads_started":1}
I20260619 04:37:16.953405 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:16.939832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43063 (local address 127.0.0.1:44244)
0619 04:37:16.945826 (+ 5994us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:16.945848 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 04:37:16.945962 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:37:16.946644 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:37:16.946650 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:37:16.946678 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:37:16.947099 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:16.947110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.948831 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.948836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:16.950920 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:16.950934 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.952586 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.952593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:16.952771 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:16.952804 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 04:37:16.952897 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 04:37:16.953022 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5873,"mutex_wait_us":38,"thread_start_us":5942,"threads_started":1}
I20260619 04:37:16.954545 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:16.939291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45341 (local address 127.0.0.1:38894)
0619 04:37:16.939768 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:16.939785 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:37:16.939888 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:37:16.947790 (+ 7902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:37:16.947795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:37:16.947820 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:37:16.948119 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:16.948129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.949697 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.949702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:16.951742 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:16.951756 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.953776 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.953782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:16.953944 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:16.953987 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 04:37:16.954078 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 04:37:16.954203 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":1,"thread_start_us":249,"threads_started":1}
I20260619 04:37:16.954584 16910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:16.939706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38894 (local address 127.0.97.2:45341)
0619 04:37:16.947221 (+ 7515us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:16.947229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:16.947275 (+ 46us) server_negotiation.cc:407] Connection header received
0619 04:37:16.947370 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:16.947377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:16.947603 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:16.947779 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:37:16.948292 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.949553 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.952067 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:16.953589 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:16.953704 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:16.953878 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:16.954132 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 04:37:16.954297 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7350,"thread_start_us":7245,"threads_started":1}
I20260619 04:37:16.954300 16569 tablet_service.cc:1467] Processing CreateTablet for tablet 810502f09dee41118f8a8bdc85d0d560 (DEFAULT_TABLE table=test-table [id=a1cd3548d1ea4680bdaf3ae03602082d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:37:16.956801 16569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810502f09dee41118f8a8bdc85d0d560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:37:16.957084 16832 tablet_service.cc:1467] Processing CreateTablet for tablet 810502f09dee41118f8a8bdc85d0d560 (DEFAULT_TABLE table=test-table [id=a1cd3548d1ea4680bdaf3ae03602082d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:37:16.957653 16701 tablet_service.cc:1467] Processing CreateTablet for tablet 810502f09dee41118f8a8bdc85d0d560 (DEFAULT_TABLE table=test-table [id=a1cd3548d1ea4680bdaf3ae03602082d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:37:16.958406 16832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810502f09dee41118f8a8bdc85d0d560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:37:16.958815 16701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810502f09dee41118f8a8bdc85d0d560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:37:16.967880 16913 tablet_bootstrap.cc:492] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f: Bootstrap starting.
I20260619 04:37:16.971388 16913 tablet_bootstrap.cc:654] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f: Neither blocks nor log segments found. Creating new log.
I20260619 04:37:16.972368 16913 log.cc:824] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f: Log is configured to *not* fsync() on all Append() calls
I20260619 04:37:16.973210 16915 tablet_bootstrap.cc:492] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5: Bootstrap starting.
I20260619 04:37:16.973757 16914 tablet_bootstrap.cc:492] T 810502f09dee41118f8a8bdc85d0d560 P 1d09e4b187424dfbafc0bbcd98b2c656: Bootstrap starting.
I20260619 04:37:16.975687 16913 tablet_bootstrap.cc:492] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f: No bootstrap required, opened a new log
I20260619 04:37:16.975934 16913 ts_tablet_manager.cc:1397] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 04:37:16.977041 16915 tablet_bootstrap.cc:654] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5: Neither blocks nor log segments found. Creating new log.
I20260619 04:37:16.977041 16914 tablet_bootstrap.cc:654] T 810502f09dee41118f8a8bdc85d0d560 P 1d09e4b187424dfbafc0bbcd98b2c656: Neither blocks nor log segments found. Creating new log.
I20260619 04:37:16.978238 16915 log.cc:824] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5: Log is configured to *not* fsync() on all Append() calls
I20260619 04:37:16.978250 16914 log.cc:824] T 810502f09dee41118f8a8bdc85d0d560 P 1d09e4b187424dfbafc0bbcd98b2c656: Log is configured to *not* fsync() on all Append() calls
I20260619 04:37:16.981130 16915 tablet_bootstrap.cc:492] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5: No bootstrap required, opened a new log
I20260619 04:37:16.981367 16915 ts_tablet_manager.cc:1397] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 04:37:16.981452 16914 tablet_bootstrap.cc:492] T 810502f09dee41118f8a8bdc85d0d560 P 1d09e4b187424dfbafc0bbcd98b2c656: No bootstrap required, opened a new log
I20260619 04:37:16.981706 16914 ts_tablet_manager.cc:1397] T 810502f09dee41118f8a8bdc85d0d560 P 1d09e4b187424dfbafc0bbcd98b2c656: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260619 04:37:16.985540 16913 raft_consensus.cc:348] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:37:16.985977 16913 raft_consensus.cc:374] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:37:16.986079 16913 raft_consensus.cc:729] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b3ffdf1457645f7a3a566928d3d3c2f, State: Initialized, Role: FOLLOWER
I20260619 04:37:16.986749 16913 consensus_queue.cc:260] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:37:16.987921 16634 heartbeater.cc:503] Master 127.0.97.62:43433 was elected leader, sending a full tablet report...
I20260619 04:37:16.988536 16913 ts_tablet_manager.cc:1428] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260619 04:37:16.989540 16915 raft_consensus.cc:348] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:37:16.989948 16915 raft_consensus.cc:374] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:37:16.990026 16915 raft_consensus.cc:729] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7197a25c401474dbcff59d51a5d69a5, State: Initialized, Role: FOLLOWER
I20260619 04:37:16.990669 16915 consensus_queue.cc:260] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:37:16.991906 16914 raft_consensus.cc:348] T 810502f09dee41118f8a8bdc85d0d560 P 1d09e4b187424dfbafc0bbcd98b2c656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:37:16.992288 16914 raft_consensus.cc:374] T 810502f09dee41118f8a8bdc85d0d560 P 1d09e4b187424dfbafc0bbcd98b2c656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:37:16.992372 16914 raft_consensus.cc:729] T 810502f09dee41118f8a8bdc85d0d560 P 1d09e4b187424dfbafc0bbcd98b2c656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d09e4b187424dfbafc0bbcd98b2c656, State: Initialized, Role: FOLLOWER
W20260619 04:37:16.993077 16635 tablet.cc:2369] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:37:16.992937 16914 consensus_queue.cc:260] T 810502f09dee41118f8a8bdc85d0d560 P 1d09e4b187424dfbafc0bbcd98b2c656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:37:16.997701 16897 heartbeater.cc:503] Master 127.0.97.62:43433 was elected leader, sending a full tablet report...
I20260619 04:37:16.998497 16915 ts_tablet_manager.cc:1428] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5: Time spent starting tablet: real 0.017s user 0.002s sys 0.015s
I20260619 04:37:16.999264 16766 heartbeater.cc:503] Master 127.0.97.62:43433 was elected leader, sending a full tablet report...
I20260619 04:37:16.999926 16914 ts_tablet_manager.cc:1428] T 810502f09dee41118f8a8bdc85d0d560 P 1d09e4b187424dfbafc0bbcd98b2c656: Time spent starting tablet: real 0.018s user 0.016s sys 0.001s
I20260619 04:37:17.105185 16920 raft_consensus.cc:482] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:37:17.105477 16920 raft_consensus.cc:504] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:37:17.107263 16920 leader_election.cc:288] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b3ffdf1457645f7a3a566928d3d3c2f (127.0.97.1:39759), 1d09e4b187424dfbafc0bbcd98b2c656 (127.0.97.2:45341)
I20260619 04:37:17.113430 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:17.107558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39759 (local address 127.0.97.3:45703)
0619 04:37:17.107805 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:17.107824 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:37:17.107943 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:37:17.108316 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:37:17.108322 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:37:17.108352 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:37:17.108615 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:17.108626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:17.109820 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:17.109824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:17.111584 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:17.111598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:17.112867 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:17.112871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:17.113025 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:17.113052 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 04:37:17.113130 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 04:37:17.113217 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":23}
I20260619 04:37:17.113550 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:17.107567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45703 (local address 127.0.97.1:39759)
0619 04:37:17.107797 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:17.107803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:17.107899 (+ 96us) server_negotiation.cc:407] Connection header received
0619 04:37:17.108083 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:17.108088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:17.108193 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:17.108286 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:37:17.108770 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:17.109682 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:17.111746 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:17.112695 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:17.112823 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:17.112916 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:17.113168 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 04:37:17.113294 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260619 04:37:17.114301 16910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:17.108266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34271 (local address 127.0.97.2:45341)
0619 04:37:17.108486 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:17.108491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:17.108893 (+ 402us) server_negotiation.cc:407] Connection header received
0619 04:37:17.108998 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:17.109003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:17.109078 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:17.109164 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:37:17.109586 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:17.110289 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:17.112291 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:17.113402 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:17.113529 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:17.113616 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:17.114026 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 04:37:17.114125 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 04:37:17.114418 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:17.108037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45341 (local address 127.0.97.3:34271)
0619 04:37:17.108806 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:17.108820 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 04:37:17.108904 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:37:17.109177 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:37:17.109180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:37:17.109194 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:37:17.109466 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:17.109473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:17.110420 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:17.110425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:17.112146 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:17.112156 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:17.113744 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:17.113750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:17.113875 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:17.113897 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 04:37:17.113968 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 04:37:17.114062 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":1,"thread_start_us":484,"threads_started":1}
I20260619 04:37:17.114557 16589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810502f09dee41118f8a8bdc85d0d560" candidate_uuid: "e7197a25c401474dbcff59d51a5d69a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" is_pre_election: true
I20260619 04:37:17.115167 16589 raft_consensus.cc:2454] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7197a25c401474dbcff59d51a5d69a5 in term 0.
I20260619 04:37:17.115342 16721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810502f09dee41118f8a8bdc85d0d560" candidate_uuid: "e7197a25c401474dbcff59d51a5d69a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" is_pre_election: true
I20260619 04:37:17.115794 16721 raft_consensus.cc:2454] T 810502f09dee41118f8a8bdc85d0d560 P 1d09e4b187424dfbafc0bbcd98b2c656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7197a25c401474dbcff59d51a5d69a5 in term 0.
I20260619 04:37:17.115841 16785 leader_election.cc:302] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b3ffdf1457645f7a3a566928d3d3c2f, e7197a25c401474dbcff59d51a5d69a5; no voters:
I20260619 04:37:17.116191 16920 raft_consensus.cc:2781] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:37:17.116315 16920 raft_consensus.cc:482] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:37:17.116415 16920 raft_consensus.cc:3037] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5 [term 0 FOLLOWER]: Advancing to term 1
W20260619 04:37:17.118289 16898 tablet.cc:2369] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:37:17.118781 16920 raft_consensus.cc:504] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:37:17.119419 16920 leader_election.cc:288] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5 [CANDIDATE]: Term 1 election: Requested vote from peers 8b3ffdf1457645f7a3a566928d3d3c2f (127.0.97.1:39759), 1d09e4b187424dfbafc0bbcd98b2c656 (127.0.97.2:45341)
I20260619 04:37:17.119875 16589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810502f09dee41118f8a8bdc85d0d560" candidate_uuid: "e7197a25c401474dbcff59d51a5d69a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f"
I20260619 04:37:17.119910 16721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810502f09dee41118f8a8bdc85d0d560" candidate_uuid: "e7197a25c401474dbcff59d51a5d69a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d09e4b187424dfbafc0bbcd98b2c656"
I20260619 04:37:17.120054 16589 raft_consensus.cc:3037] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:37:17.120085 16721 raft_consensus.cc:3037] T 810502f09dee41118f8a8bdc85d0d560 P 1d09e4b187424dfbafc0bbcd98b2c656 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:37:17.122172 16589 raft_consensus.cc:2454] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7197a25c401474dbcff59d51a5d69a5 in term 1.
I20260619 04:37:17.122278 16721 raft_consensus.cc:2454] T 810502f09dee41118f8a8bdc85d0d560 P 1d09e4b187424dfbafc0bbcd98b2c656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7197a25c401474dbcff59d51a5d69a5 in term 1.
I20260619 04:37:17.122545 16785 leader_election.cc:302] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b3ffdf1457645f7a3a566928d3d3c2f, e7197a25c401474dbcff59d51a5d69a5; no voters:
I20260619 04:37:17.122814 16920 raft_consensus.cc:2781] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:37:17.123205 16920 raft_consensus.cc:686] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5 [term 1 LEADER]: Becoming Leader. State: Replica: e7197a25c401474dbcff59d51a5d69a5, State: Running, Role: LEADER
I20260619 04:37:17.123706 16920 consensus_queue.cc:237] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:37:17.128316 16448 catalog_manager.cc:5495] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7197a25c401474dbcff59d51a5d69a5 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e7197a25c401474dbcff59d51a5d69a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } health_report { overall_health: HEALTHY } } }
I20260619 04:37:17.144011 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:17.139326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44260 (local address 127.0.97.3:43063)
0619 04:37:17.139491 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:17.139496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:17.139511 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:37:17.139651 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:17.139655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:17.139737 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:17.139853 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:37:17.140140 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:17.140883 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:17.142128 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:17.142509 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:17.142599 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:17.143031 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:37:17.143521 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:37:17.143634 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:17.143751 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 04:37:17.143815 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 04:37:17.151770 16721 raft_consensus.cc:1261] T 810502f09dee41118f8a8bdc85d0d560 P 1d09e4b187424dfbafc0bbcd98b2c656 [term 1 FOLLOWER]: Refusing update from remote peer e7197a25c401474dbcff59d51a5d69a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:37:17.151778 16589 raft_consensus.cc:1261] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Refusing update from remote peer e7197a25c401474dbcff59d51a5d69a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:37:17.152658 16923 consensus_queue.cc:1035] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:37:17.152984 16920 consensus_queue.cc:1035] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:37:17.164793 16927 mvcc.cc:205] Tried to move back new op lower bound from 7298432356964036608 to 7298432356862738432. Current Snapshot: MvccSnapshot[applied={T|T < 7298432356964036608}]
I20260619 04:37:17.176843 16449 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:37:17.177145 16449 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:37:17.180568 16449 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8: Sending DeleteTablet for 3 replicas of tablet 810502f09dee41118f8a8bdc85d0d560
I20260619 04:37:17.181653 16569 tablet_service.cc:1514] Processing DeleteTablet for tablet 810502f09dee41118f8a8bdc85d0d560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:37:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49620
I20260619 04:37:17.181785 16701 tablet_service.cc:1514] Processing DeleteTablet for tablet 810502f09dee41118f8a8bdc85d0d560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:37:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38894
I20260619 04:37:17.181972 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 16486
I20260619 04:37:17.182441 16832 tablet_service.cc:1514] Processing DeleteTablet for tablet 810502f09dee41118f8a8bdc85d0d560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:37:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44244
I20260619 04:37:17.182579 16935 tablet_replica.cc:330] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f: stopping tablet replica
I20260619 04:37:17.184239 16937 tablet_replica.cc:330] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5: stopping tablet replica
I20260619 04:37:17.184648 16937 raft_consensus.cc:2229] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:37:17.185298 16937 raft_consensus.cc:2258] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:37:17.183584 16935 raft_consensus.cc:2229] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:37:17.186190 16935 raft_consensus.cc:2258] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:37:17.187623 16936 tablet_replica.cc:330] T 810502f09dee41118f8a8bdc85d0d560 P 1d09e4b187424dfbafc0bbcd98b2c656: stopping tablet replica
I20260619 04:37:17.187988 16936 raft_consensus.cc:2229] T 810502f09dee41118f8a8bdc85d0d560 P 1d09e4b187424dfbafc0bbcd98b2c656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:37:17.189468 16936 raft_consensus.cc:2258] T 810502f09dee41118f8a8bdc85d0d560 P 1d09e4b187424dfbafc0bbcd98b2c656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:37:17.189733 16937 ts_tablet_manager.cc:1905] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:37:17.191327 16936 ts_tablet_manager.cc:1905] T 810502f09dee41118f8a8bdc85d0d560 P 1d09e4b187424dfbafc0bbcd98b2c656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:37:17.197059 16937 ts_tablet_manager.cc:1918] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:37:17.197338 16937 log.cc:1196] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/810502f09dee41118f8a8bdc85d0d560
I20260619 04:37:17.197966 16936 ts_tablet_manager.cc:1918] T 810502f09dee41118f8a8bdc85d0d560 P 1d09e4b187424dfbafc0bbcd98b2c656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:37:17.198118 16936 log.cc:1196] T 810502f09dee41118f8a8bdc85d0d560 P 1d09e4b187424dfbafc0bbcd98b2c656: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/810502f09dee41118f8a8bdc85d0d560
I20260619 04:37:17.198127 16937 ts_tablet_manager.cc:1939] T 810502f09dee41118f8a8bdc85d0d560 P e7197a25c401474dbcff59d51a5d69a5: Deleting consensus metadata
I20260619 04:37:17.198609 16936 ts_tablet_manager.cc:1939] T 810502f09dee41118f8a8bdc85d0d560 P 1d09e4b187424dfbafc0bbcd98b2c656: Deleting consensus metadata
I20260619 04:37:17.199667 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:17.183166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49626 (local address 127.0.97.1:39759)
0619 04:37:17.183341 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:17.183347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:17.183365 (+ 18us) server_negotiation.cc:407] Connection header received
0619 04:37:17.183418 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:17.183423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:17.183527 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:17.183635 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:37:17.185464 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:17.186635 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:17.188211 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:17.188844 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:17.188968 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:17.192152 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:37:17.192165 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:37:17.192181 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:37:17.192218 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:37:17.195646 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:17.196525 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:17.196534 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:17.196538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:17.196677 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:17.197151 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:17.197156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:17.197159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:17.197666 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:37:17.197938 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:17.199125 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0619 04:37:17.199333 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 04:37:17.200302 16629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:37:17.200320 16433 catalog_manager.cc:4841] TS e7197a25c401474dbcff59d51a5d69a5 (127.0.97.3:43063): tablet 810502f09dee41118f8a8bdc85d0d560 (table test-table [id=a1cd3548d1ea4680bdaf3ae03602082d]) successfully deleted
I20260619 04:37:17.200677 16433 catalog_manager.cc:4841] TS 1d09e4b187424dfbafc0bbcd98b2c656 (127.0.97.2:45341): tablet 810502f09dee41118f8a8bdc85d0d560 (table test-table [id=a1cd3548d1ea4680bdaf3ae03602082d]) successfully deleted
I20260619 04:37:17.309759 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 16486
W20260619 04:37:17.324142 16433 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39759 (error 108)
I20260619 04:37:17.324409 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 16638
I20260619 04:37:17.326423 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:17.325887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39759 (local address 127.0.0.1:49634)
0619 04:37:17.326127 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:17.326245 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260619 04:37:17.326671 16433 catalog_manager.cc:4584] TS 8b3ffdf1457645f7a3a566928d3d3c2f (127.0.97.1:39759): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 810502f09dee41118f8a8bdc85d0d560: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39759: connect: Connection refused (error 111)
I20260619 04:37:17.337831 16910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:17.325244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38908 (local address 127.0.97.2:45341)
0619 04:37:17.325441 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:17.325447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:17.325571 (+ 124us) server_negotiation.cc:407] Connection header received
0619 04:37:17.325760 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:17.325765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:17.325840 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:17.325925 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:37:17.327309 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:17.328137 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:17.329625 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:17.330019 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:17.330125 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:17.332557 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:37:17.332570 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:37:17.332584 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:37:17.332615 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:37:17.335278 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:17.335900 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:17.335909 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:17.335911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:17.336044 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:17.336378 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:17.336383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:17.336386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:17.336755 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:37:17.337042 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:17.337308 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 04:37:17.337506 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:37:17.338404 16761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:37:17.380388 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:17.379746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39759 (local address 127.0.0.1:49638)
0619 04:37:17.380015 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:17.380134 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 04:37:17.381631 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:17.381174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39759 (local address 127.0.0.1:49640)
0619 04:37:17.381340 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:17.381489 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 04:37:17.426750 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 16638
I20260619 04:37:17.442221 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:17.441789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39759 (local address 127.0.0.1:49650)
0619 04:37:17.441992 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:17.442079 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 04:37:17.443195 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:17.442857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39759 (local address 127.0.0.1:49666)
0619 04:37:17.443015 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:17.443084 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 04:37:17.446766 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 16769
I20260619 04:37:17.462296 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:17.447439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44262 (local address 127.0.97.3:43063)
0619 04:37:17.447585 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:17.447590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:17.447609 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:37:17.447811 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:17.447815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:17.447877 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:17.447955 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:37:17.449711 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:17.450566 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:17.452425 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:17.453013 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:17.453109 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:17.456362 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:37:17.456374 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:37:17.456393 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:37:17.456427 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:37:17.459374 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:17.460153 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:17.460160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:17.460162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:17.460314 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:17.460860 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:17.460863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:17.460865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:17.461251 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:37:17.461482 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:17.461836 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 04:37:17.462005 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 04:37:17.462965 16892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:37:17.524020 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:17.523515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39759 (local address 127.0.0.1:49682)
0619 04:37:17.523745 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:17.523851 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 04:37:17.525226 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:17.524766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39759 (local address 127.0.0.1:49688)
0619 04:37:17.524996 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:17.525086 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260619 04:37:17.584247 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 16769
I20260619 04:37:17.599807 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 16416
I20260619 04:37:17.600878 16479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:37:17.714205 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:17.713596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39759 (local address 127.0.0.1:49696)
0619 04:37:17.713821 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:17.713945 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 04:37:17.714506 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 16416
I20260619 04:37:37.732647 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43433
--webserver_interface=127.0.97.62
--webserver_port=44781
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:37:38.050721 16943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:37:38.051029 16943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:37:38.051121 16943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:37:38.058493 16943 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 04:37:38.058604 16943 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 04:37:38.058640 16943 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 04:37:38.058689 16943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:37:38.058748 16943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:37:38.058812 16943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:37:38.058856 16943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:37:38.058907 16943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:37:38.068849 16943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:37:38.070066 16943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:37:38.071309 16943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:37:38.082486 16951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:38.082549 16949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:38.082523 16948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:37:38.083050 16943 server_base.cc:1034] running on GCE node
I20260619 04:37:38.083812 16943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:37:38.085248 16943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:37:38.086478 16943 hybrid_clock.cc:648] HybridClock initialized: now 1781843858086406 us; error 51 us; skew 500 ppm
I20260619 04:37:38.086814 16943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:37:38 dist-test-slave-n3tq krb5kdc[15866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:37:38.089931 16943 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:37:38.091355 16943 webserver.cc:460] Webserver started at http://127.0.97.62:44781/ using document root <none> and password file <none>
I20260619 04:37:38.091996 16943 fs_manager.cc:359] Metadata directory not provided
I20260619 04:37:38.092077 16943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:37:38.097383 16943 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:37:38.100087 16958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:38.101430 16943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:37:38.101581 16943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "2dfec6738f1d4b63b701d10fcc07efd8"
format_stamp: "Formatted at 2026-06-19 04:36:52 on dist-test-slave-n3tq"
server_key: "2afdf9a6fbbcb3fe64c6ada60e092168"
server_key_iv: "ed05cb37290d75868864b605aacd2398"
server_key_version: "encryptionkey@0"
I20260619 04:37:38.102490 16943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:37:38.119817 16943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:37:38.123639 16943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:37:38.123915 16943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:37:38.140245 16943 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43433
I20260619 04:37:38.140283 17009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43433 every 8 connection(s)
I20260619 04:37:38.141716 16943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:37:38.144032 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 16943
I20260619 04:37:38.145056 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39759
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:43433
--webserver_port=33197
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:37:38.150348 17010 sys_catalog.cc:263] Verifying existing consensus state
I20260619 04:37:38.154528 17010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8: Bootstrap starting.
I20260619 04:37:38.174350 17010 log.cc:824] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8: Log is configured to *not* fsync() on all Append() calls
I20260619 04:37:38.191963 17010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 04:37:38.192691 17010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8: Bootstrap complete.
I20260619 04:37:38.203162 17010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } }
I20260619 04:37:38.204047 17010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dfec6738f1d4b63b701d10fcc07efd8, State: Initialized, Role: FOLLOWER
I20260619 04:37:38.204656 17010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } }
I20260619 04:37:38.204855 17010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:37:38.204977 17010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:37:38.205123 17010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 2 FOLLOWER]: Advancing to term 3
I20260619 04:37:38.208986 17010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } }
I20260619 04:37:38.209456 17010 leader_election.cc:302] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dfec6738f1d4b63b701d10fcc07efd8; no voters:
I20260619 04:37:38.210063 17010 leader_election.cc:288] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 04:37:38.210203 17014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 04:37:38.212179 17010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:37:38.217170 17014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [term 3 LEADER]: Becoming Leader. State: Replica: 2dfec6738f1d4b63b701d10fcc07efd8, State: Running, Role: LEADER
I20260619 04:37:38.217844 17014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } }
I20260619 04:37:38.220541 17015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } } }
I20260619 04:37:38.220954 17015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [sys.catalog]: This master's current role is: LEADER
I20260619 04:37:38.221876 17029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:37:38.221776 17016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dfec6738f1d4b63b701d10fcc07efd8. Latest consensus state: current_term: 3 leader_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfec6738f1d4b63b701d10fcc07efd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } } }
I20260619 04:37:38.222711 17016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8 [sys.catalog]: This master's current role is: LEADER
I20260619 04:37:38.235227 17029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:37:38.241462 17029 catalog_manager.cc:1259] Loaded cluster ID: 91cdcaea8b9a4f9395e552a490828bbd
I20260619 04:37:38.241609 17029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:37:38.245584 17029 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:37:38.247246 17029 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8: Loaded TSK: 0
I20260619 04:37:38.247392 17029 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2dfec6738f1d4b63b701d10fcc07efd8: Loaded TSK: 1
F20260619 04:37:38.248559 17029 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f70517c49ad google::LogMessage::Fail() at ??:0
@ 0x7f70517c788a google::LogMessage::SendToLog() at ??:0
@ 0x7f70517c44de google::LogMessage::Flush() at ??:0
@ 0x7f70517c5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f70555287d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f705551773e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f70671f71c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f70671f6783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f70672822a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f70672820c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f70672cc5e9 std::function<>::operator()() at ??:0
@ 0x7f70671facdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f70671f8c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7067280610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7067280439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7054410345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f705441391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f7054413799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f70543eeeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f70576c86db start_thread at ??:0
@ 0x7f705065e61f clone at ??:0
W20260619 04:37:38.522073 17012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:37:38.522365 17012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:37:38.522459 17012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:37:38.529850 17012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:37:38.529968 17012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:37:38.530081 17012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:37:38.539985 17012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39759
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=33197
--tserver_master_addrs=127.0.97.62:43433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:37:38.541280 17012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:37:38.542546 17012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:37:38.555361 17061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:38.555348 17058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:38.555657 17059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:37:38.555940 17012 server_base.cc:1034] running on GCE node
I20260619 04:37:38.556675 17012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:37:38.557595 17012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:37:38.558802 17012 hybrid_clock.cc:648] HybridClock initialized: now 1781843858558747 us; error 47 us; skew 500 ppm
I20260619 04:37:38.559111 17012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:37:38 dist-test-slave-n3tq krb5kdc[15866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:37:38.562184 17012 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:37:38.563556 17012 webserver.cc:460] Webserver started at http://127.0.97.1:33197/ using document root <none> and password file <none>
I20260619 04:37:38.564244 17012 fs_manager.cc:359] Metadata directory not provided
I20260619 04:37:38.564352 17012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:37:38.569595 17012 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:37:38.572438 17068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:38.573828 17012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:37:38.573971 17012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "8b3ffdf1457645f7a3a566928d3d3c2f"
format_stamp: "Formatted at 2026-06-19 04:36:52 on dist-test-slave-n3tq"
server_key: "0560416ef9b44134c8bf5e36ae4bcc1a"
server_key_iv: "dc214dc89012c44732d8bc4f291543eb"
server_key_version: "encryptionkey@0"
I20260619 04:37:38.574925 17012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:37:38.598608 17012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:37:38.602886 17012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:37:38.603142 17012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:37:38.604108 17012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:37:38.605911 17075 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260619 04:37:38.611505 17012 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 04:37:38.611624 17012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260619 04:37:38.611729 17012 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 04:37:38.615319 17012 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 04:37:38.615419 17012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.001s
I20260619 04:37:38.615837 17075 tablet_bootstrap.cc:492] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f: Bootstrap starting.
I20260619 04:37:38.624825 17075 log.cc:824] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f: Log is configured to *not* fsync() on all Append() calls
I20260619 04:37:38.639240 17075 tablet_bootstrap.cc:492] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 04:37:38.640055 17075 tablet_bootstrap.cc:492] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f: Bootstrap complete.
I20260619 04:37:38.640662 17075 ts_tablet_manager.cc:1397] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f: Time spent bootstrapping tablet: real 0.025s user 0.022s sys 0.002s
I20260619 04:37:38.650520 17075 raft_consensus.cc:348] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:37:38.652063 17075 raft_consensus.cc:729] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b3ffdf1457645f7a3a566928d3d3c2f, State: Initialized, Role: FOLLOWER
I20260619 04:37:38.653208 17075 consensus_queue.cc:260] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:37:38.654906 17075 ts_tablet_manager.cc:1428] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
I20260619 04:37:38.655982 17012 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39759
I20260619 04:37:38.656037 17183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39759 every 8 connection(s)
I20260619 04:37:38.657215 17012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:37:38.661026 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 17012
I20260619 04:37:38.661621 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45341
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:43433
--webserver_port=35527
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:37:38.661814 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:38.659776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.1:45749)
0619 04:37:38.660633 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:38.660866 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":338,"threads_started":1}
W20260619 04:37:38.662508 17184 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43433 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:43433: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
I20260619 04:37:38.663303 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:38.662986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.1:56535)
0619 04:37:38.663129 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:38.663199 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 04:37:38.664098 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:38.663803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.1:48603)
0619 04:37:38.663919 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:38.663987 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 04:37:38.664316 17184 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 04:37:38.975468 17187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:37:38.975744 17187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:37:38.975867 17187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:37:38.983170 17187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:37:38.983273 17187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:37:38.983417 17187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:37:38.992950 17187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35527
--tserver_master_addrs=127.0.97.62:43433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:37:38.994182 17187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:37:38.995438 17187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:37:39.007874 17193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:39.007834 17195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:39.008486 17192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:37:39.008621 17187 server_base.cc:1034] running on GCE node
I20260619 04:37:39.009236 17187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:37:39.010304 17187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:37:39.011521 17187 hybrid_clock.cc:648] HybridClock initialized: now 1781843859011465 us; error 45 us; skew 500 ppm
I20260619 04:37:39.011833 17187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:37:39 dist-test-slave-n3tq krb5kdc[15866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:37:39.014889 17187 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:37:39.016299 17187 webserver.cc:460] Webserver started at http://127.0.97.2:35527/ using document root <none> and password file <none>
I20260619 04:37:39.016995 17187 fs_manager.cc:359] Metadata directory not provided
I20260619 04:37:39.017112 17187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:37:39.022392 17187 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 04:37:39.025102 17202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:39.026383 17187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:37:39.026531 17187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "1d09e4b187424dfbafc0bbcd98b2c656"
format_stamp: "Formatted at 2026-06-19 04:36:53 on dist-test-slave-n3tq"
server_key: "ceef8873bba095c963df495a9408ff23"
server_key_iv: "72b3734290901d47f8e5f7da4915fdb0"
server_key_version: "encryptionkey@0"
I20260619 04:37:39.027488 17187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:37:39.047222 17187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:37:39.051278 17187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:37:39.051517 17187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:37:39.052502 17187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:37:39.053747 17187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:37:39.053853 17187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:39.053952 17187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:37:39.054016 17187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:39.083496 17187 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45341
I20260619 04:37:39.083559 17314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45341 every 8 connection(s)
I20260619 04:37:39.084940 17187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:37:39.089534 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:39.087706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.2:38367)
0619 04:37:39.088424 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:39.088656 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":279,"threads_started":1}
W20260619 04:37:39.090158 17315 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43433 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:43433: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
I20260619 04:37:39.090885 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:39.090577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.2:39237)
0619 04:37:39.090722 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:39.090788 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 04:37:39.091581 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:39.091316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.2:56857)
0619 04:37:39.091414 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:39.091483 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 04:37:39.091789 17315 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 04:37:39.092847 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 17187
I20260619 04:37:39.093422 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43063
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:43433
--webserver_port=42531
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:37:39.400893 17318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:37:39.401186 17318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:37:39.401304 17318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:37:39.408489 17318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:37:39.408623 17318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:37:39.408779 17318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:37:39.418401 17318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=42531
--tserver_master_addrs=127.0.97.62:43433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:37:39.419653 17318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:37:39.420998 17318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:37:39.433362 17323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:39.433275 17324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:39.433293 17326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:37:39.434515 17318 server_base.cc:1034] running on GCE node
I20260619 04:37:39.435108 17318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:37:39.436002 17318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:37:39.437189 17318 hybrid_clock.cc:648] HybridClock initialized: now 1781843859437137 us; error 44 us; skew 500 ppm
I20260619 04:37:39.437464 17318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:37:39 dist-test-slave-n3tq krb5kdc[15866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:37:39.440552 17318 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:37:39.442230 17318 webserver.cc:460] Webserver started at http://127.0.97.3:42531/ using document root <none> and password file <none>
I20260619 04:37:39.442850 17318 fs_manager.cc:359] Metadata directory not provided
I20260619 04:37:39.442945 17318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:37:39.448062 17318 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:37:39.451924 17333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:39.453281 17318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 04:37:39.453403 17318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "e7197a25c401474dbcff59d51a5d69a5"
format_stamp: "Formatted at 2026-06-19 04:36:53 on dist-test-slave-n3tq"
server_key: "15a687eb8db89a2d703cb42100785b4e"
server_key_iv: "717437a6b1d0ff045e0c89f8b26c28cb"
server_key_version: "encryptionkey@0"
I20260619 04:37:39.454301 17318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:37:39.467828 17318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:37:39.471728 17318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:37:39.471944 17318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:37:39.472855 17318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:37:39.474185 17318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:37:39.474269 17318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:39.474344 17318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:37:39.474395 17318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:39.503518 17318 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43063
I20260619 04:37:39.503551 17445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43063 every 8 connection(s)
I20260619 04:37:39.505003 17318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:37:39.509670 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:39.507874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.3:56915)
0619 04:37:39.508613 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:39.508803 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":268,"threads_started":1}
W20260619 04:37:39.510352 17446 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43433 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:43433: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
I20260619 04:37:39.511090 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:39.510805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.3:36031)
0619 04:37:39.510926 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:39.510990 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 04:37:39.511830 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:39.511535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.3:36027)
0619 04:37:39.511665 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:39.511734 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 04:37:39.512027 17446 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 04:37:39.513882 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 17318
I20260619 04:37:39.665638 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:39.664882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.1:41239)
0619 04:37:39.665299 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:39.665378 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":188,"threads_started":1}
I20260619 04:37:39.885205 17451 raft_consensus.cc:482] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:37:39.885473 17451 raft_consensus.cc:504] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:37:39.887071 17451 leader_election.cc:288] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d09e4b187424dfbafc0bbcd98b2c656 (127.0.97.2:45341), e7197a25c401474dbcff59d51a5d69a5 (127.0.97.3:43063)
Jun 19 04:37:39 dist-test-slave-n3tq krb5kdc[15866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
Jun 19 04:37:39 dist-test-slave-n3tq krb5kdc[15866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 04:37:39.908330 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:39.887368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45341 (local address 127.0.97.1:44859)
0619 04:37:39.887559 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:39.887607 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 04:37:39.888206 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:37:39.891674 (+ 3468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:37:39.891684 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:37:39.892585 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:37:39.893263 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:39.893283 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:39.894784 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:39.894788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:39.896280 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:39.896293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:39.897482 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:39.897488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:39.897788 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:39.898520 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:37:39.898549 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:37:39.900988 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:37:39.904337 (+ 3349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:37:39.904346 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:37:39.904361 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:37:39.905076 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:37:39.905514 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:37:39.905517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:37:39.905520 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:37:39.905725 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:37:39.906647 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:37:39.906653 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:37:39.907014 (+ 361us) client_negotiation.cc:769] Sending connection context
0619 04:37:39.907356 (+ 342us) client_negotiation.cc:240] Negotiation successful
0619 04:37:39.907883 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":188,"spinlock_wait_cycles":14080}
I20260619 04:37:39.908437 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:39.887622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44859 (local address 127.0.97.2:45341)
0619 04:37:39.889245 (+ 1623us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:39.889263 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:39.889306 (+ 43us) server_negotiation.cc:407] Connection header received
0619 04:37:39.890823 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:39.890832 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:39.891212 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:39.891529 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:37:39.893445 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:39.894597 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:39.896401 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:39.897288 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:39.897455 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:39.901255 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:37:39.901263 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:37:39.901277 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:37:39.901317 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:37:39.904194 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:39.905217 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:39.905224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:39.905230 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:39.905393 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:39.905824 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:39.905827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:39.905829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:39.906502 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:37:39.906689 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:39.907642 (+ 953us) server_negotiation.cc:299] Negotiation successful
0619 04:37:39.908036 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1434,"thread_start_us":1308,"threads_started":1}
I20260619 04:37:39.908434 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:39.888150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43297 (local address 127.0.97.3:43063)
0619 04:37:39.889553 (+ 1403us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:39.889563 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:39.889596 (+ 33us) server_negotiation.cc:407] Connection header received
0619 04:37:39.890990 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:39.890998 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:39.891330 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:39.891643 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:37:39.893444 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:39.894608 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:39.896469 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:39.897346 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:39.897508 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:39.901352 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:37:39.901360 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:37:39.901372 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:37:39.901407 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:37:39.904296 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:39.905537 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:39.905543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:39.905549 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:39.905690 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:39.906090 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:39.906095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:39.906098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:39.906699 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:37:39.906839 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:39.907620 (+ 781us) server_negotiation.cc:299] Negotiation successful
0619 04:37:39.908034 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1252,"thread_start_us":1109,"threads_started":1}
I20260619 04:37:39.908380 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:39.887732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43063 (local address 127.0.97.1:43297)
0619 04:37:39.888996 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:39.889013 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:37:39.889120 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:37:39.892087 (+ 2967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:37:39.892092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:37:39.892729 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:37:39.893262 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:39.893284 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:39.894774 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:39.894778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:39.896283 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:39.896297 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:39.897539 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:39.897546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:39.897782 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:39.898587 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:37:39.898603 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:37:39.901118 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:37:39.904391 (+ 3273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:37:39.904395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:37:39.904398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:37:39.905076 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:37:39.905800 (+ 724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:37:39.905804 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:37:39.905807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:37:39.905939 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:37:39.906821 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:37:39.906826 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:37:39.907014 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 04:37:39.907356 (+ 342us) client_negotiation.cc:240] Negotiation successful
0619 04:37:39.908027 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1135,"mutex_wait_us":202,"spinlock_wait_cycles":14336,"thread_start_us":949,"threads_started":1}
W20260619 04:37:39.911731 17069 leader_election.cc:341] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d09e4b187424dfbafc0bbcd98b2c656 (127.0.97.2:45341): Not found: Tablet not found: 810502f09dee41118f8a8bdc85d0d560
W20260619 04:37:39.912029 17069 leader_election.cc:341] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7197a25c401474dbcff59d51a5d69a5 (127.0.97.3:43063): Not found: Tablet not found: 810502f09dee41118f8a8bdc85d0d560
I20260619 04:37:39.912151 17069 leader_election.cc:302] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b3ffdf1457645f7a3a566928d3d3c2f; no voters: 1d09e4b187424dfbafc0bbcd98b2c656, e7197a25c401474dbcff59d51a5d69a5
I20260619 04:37:39.912580 17451 raft_consensus.cc:2726] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 04:37:40.093238 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:40.092216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.2:57951)
0619 04:37:40.092721 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:40.092852 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":257,"threads_started":1}
I20260619 04:37:40.513226 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:40.512559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.3:37039)
0619 04:37:40.512929 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:40.513042 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":177,"threads_started":1}
I20260619 04:37:40.667112 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:40.666427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.1:54205)
0619 04:37:40.666869 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:40.666947 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":229,"threads_started":1}
I20260619 04:37:41.094830 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:41.094153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.2:58045)
0619 04:37:41.094594 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:41.094677 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":217,"threads_started":1}
I20260619 04:37:41.514815 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:41.514077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.3:55099)
0619 04:37:41.514560 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:41.514641 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":272,"threads_started":1}
I20260619 04:37:41.668712 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:41.667969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.1:57937)
0619 04:37:41.668411 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:41.668533 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":200,"threads_started":1}
I20260619 04:37:41.976297 17461 raft_consensus.cc:482] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:37:41.976524 17461 raft_consensus.cc:504] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:37:41.977172 17461 leader_election.cc:288] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d09e4b187424dfbafc0bbcd98b2c656 (127.0.97.2:45341), e7197a25c401474dbcff59d51a5d69a5 (127.0.97.3:43063)
W20260619 04:37:41.978188 17069 leader_election.cc:341] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7197a25c401474dbcff59d51a5d69a5 (127.0.97.3:43063): Not found: Tablet not found: 810502f09dee41118f8a8bdc85d0d560
W20260619 04:37:41.978417 17069 leader_election.cc:341] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d09e4b187424dfbafc0bbcd98b2c656 (127.0.97.2:45341): Not found: Tablet not found: 810502f09dee41118f8a8bdc85d0d560
I20260619 04:37:41.978499 17069 leader_election.cc:302] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b3ffdf1457645f7a3a566928d3d3c2f; no voters: 1d09e4b187424dfbafc0bbcd98b2c656, e7197a25c401474dbcff59d51a5d69a5
I20260619 04:37:41.978757 17461 raft_consensus.cc:2726] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 04:37:42.096324 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:42.095653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.2:49113)
0619 04:37:42.096063 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:42.096164 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":204,"threads_started":1}
I20260619 04:37:42.516342 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:42.515639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.3:49915)
0619 04:37:42.516083 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:42.516163 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":225,"threads_started":1}
I20260619 04:37:42.670245 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:42.669554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.1:60643)
0619 04:37:42.669990 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:42.670080 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":229,"threads_started":1}
I20260619 04:37:43.097921 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:43.097213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.2:42585)
0619 04:37:43.097648 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:43.097732 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":198,"threads_started":1}
I20260619 04:37:43.517952 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:43.517173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.3:47313)
0619 04:37:43.517658 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:43.517748 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":250,"threads_started":1}
I20260619 04:37:43.672022 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:43.671242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.1:47041)
0619 04:37:43.671747 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:43.671843 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":249,"threads_started":1}
I20260619 04:37:44.099529 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:44.098802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.2:58807)
0619 04:37:44.099265 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:44.099347 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":251,"threads_started":1}
I20260619 04:37:44.519531 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:44.518826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.3:57595)
0619 04:37:44.519281 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:44.519377 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":218,"threads_started":1}
I20260619 04:37:44.662336 17470 raft_consensus.cc:482] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:37:44.662518 17470 raft_consensus.cc:504] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:37:44.663151 17470 leader_election.cc:288] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d09e4b187424dfbafc0bbcd98b2c656 (127.0.97.2:45341), e7197a25c401474dbcff59d51a5d69a5 (127.0.97.3:43063)
W20260619 04:37:44.664113 17069 leader_election.cc:341] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7197a25c401474dbcff59d51a5d69a5 (127.0.97.3:43063): Not found: Tablet not found: 810502f09dee41118f8a8bdc85d0d560
W20260619 04:37:44.664304 17069 leader_election.cc:341] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d09e4b187424dfbafc0bbcd98b2c656 (127.0.97.2:45341): Not found: Tablet not found: 810502f09dee41118f8a8bdc85d0d560
I20260619 04:37:44.664407 17069 leader_election.cc:302] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b3ffdf1457645f7a3a566928d3d3c2f; no voters: 1d09e4b187424dfbafc0bbcd98b2c656, e7197a25c401474dbcff59d51a5d69a5
I20260619 04:37:44.664691 17470 raft_consensus.cc:2726] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 04:37:44.673707 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:44.672948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.1:59637)
0619 04:37:44.673432 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:44.673526 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":248,"threads_started":1}
I20260619 04:37:45.101368 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:45.100552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.2:57285)
0619 04:37:45.101086 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:45.101177 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":291,"threads_started":1}
I20260619 04:37:45.521028 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:45.520328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.3:40317)
0619 04:37:45.520776 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:45.520855 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":246,"threads_started":1}
I20260619 04:37:45.675225 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:45.674528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.1:37881)
0619 04:37:45.674965 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:45.675047 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":219,"threads_started":1}
I20260619 04:37:46.103034 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:46.102234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.2:47747)
0619 04:37:46.102764 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:46.102854 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":219,"threads_started":1}
I20260619 04:37:46.522718 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:46.521885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.3:58017)
0619 04:37:46.522412 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:46.522499 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":245,"threads_started":1}
I20260619 04:37:46.676843 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:46.676088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.1:51891)
0619 04:37:46.676600 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:46.676681 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":258,"threads_started":1}
I20260619 04:37:47.104988 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:47.104069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.2:41807)
0619 04:37:47.104659 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:47.104780 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":262,"threads_started":1}
I20260619 04:37:47.524267 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:47.523610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.3:49197)
0619 04:37:47.524010 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:47.524089 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":200,"threads_started":1}
I20260619 04:37:47.678399 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:47.677670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.1:43869)
0619 04:37:47.678115 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:47.678206 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":221,"threads_started":1}
I20260619 04:37:48.035321 17481 raft_consensus.cc:482] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:37:48.035521 17481 raft_consensus.cc:504] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:37:48.036069 17481 leader_election.cc:288] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d09e4b187424dfbafc0bbcd98b2c656 (127.0.97.2:45341), e7197a25c401474dbcff59d51a5d69a5 (127.0.97.3:43063)
W20260619 04:37:48.036967 17069 leader_election.cc:341] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d09e4b187424dfbafc0bbcd98b2c656 (127.0.97.2:45341): Not found: Tablet not found: 810502f09dee41118f8a8bdc85d0d560
W20260619 04:37:48.037171 17069 leader_election.cc:341] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7197a25c401474dbcff59d51a5d69a5 (127.0.97.3:43063): Not found: Tablet not found: 810502f09dee41118f8a8bdc85d0d560
I20260619 04:37:48.037254 17069 leader_election.cc:302] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b3ffdf1457645f7a3a566928d3d3c2f; no voters: 1d09e4b187424dfbafc0bbcd98b2c656, e7197a25c401474dbcff59d51a5d69a5
I20260619 04:37:48.037526 17481 raft_consensus.cc:2726] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 04:37:48.106652 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:48.105879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.2:33695)
0619 04:37:48.106399 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:48.106485 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":247,"threads_started":1}
I20260619 04:37:48.525789 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:48.525114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.3:47269)
0619 04:37:48.525548 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:48.525634 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":202,"threads_started":1}
I20260619 04:37:48.679996 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:48.679297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.1:38275)
0619 04:37:48.679746 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:48.679830 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":231,"threads_started":1}
I20260619 04:37:49.108430 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:49.107663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.2:33053)
0619 04:37:49.108142 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:49.108238 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":250,"threads_started":1}
I20260619 04:37:49.527410 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:49.526715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.3:38441)
0619 04:37:49.527130 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:49.527211 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":212,"threads_started":1}
I20260619 04:37:49.681689 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:49.680928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.1:46147)
0619 04:37:49.681407 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:49.681510 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":235,"threads_started":1}
I20260619 04:37:50.110020 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:50.109312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.2:55073)
0619 04:37:50.109777 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:50.109863 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":222,"threads_started":1}
I20260619 04:37:50.529685 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:50.528385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.3:33823)
0619 04:37:50.529453 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:50.529533 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":955,"thread_start_us":808,"threads_started":1}
I20260619 04:37:50.683194 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:50.682519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.1:60239)
0619 04:37:50.682938 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:50.683028 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":215,"threads_started":1}
I20260619 04:37:51.111835 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:51.111030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.2:40687)
0619 04:37:51.111555 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:51.111645 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":282,"threads_started":1}
I20260619 04:37:51.531415 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:51.530629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.3:45393)
0619 04:37:51.531142 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:51.531232 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":274,"threads_started":1}
I20260619 04:37:51.684880 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:51.684142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.1:33155)
0619 04:37:51.684603 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:51.684701 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":202,"threads_started":1}
I20260619 04:37:52.113452 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:52.112744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.2:36639)
0619 04:37:52.113196 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:52.113274 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":223,"threads_started":1}
I20260619 04:37:52.533061 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:52.532333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.3:55375)
0619 04:37:52.532805 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:52.532888 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":239,"threads_started":1}
I20260619 04:37:52.686391 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:52.685704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.1:45525)
0619 04:37:52.686129 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:52.686210 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":214,"threads_started":1}
I20260619 04:37:53.070163 17497 raft_consensus.cc:482] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:37:53.070348 17497 raft_consensus.cc:504] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ffdf1457645f7a3a566928d3d3c2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39759 } } peers { permanent_uuid: "1d09e4b187424dfbafc0bbcd98b2c656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45341 } } peers { permanent_uuid: "e7197a25c401474dbcff59d51a5d69a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43063 } }
I20260619 04:37:53.071028 17497 leader_election.cc:288] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d09e4b187424dfbafc0bbcd98b2c656 (127.0.97.2:45341), e7197a25c401474dbcff59d51a5d69a5 (127.0.97.3:43063)
W20260619 04:37:53.071903 17069 leader_election.cc:341] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d09e4b187424dfbafc0bbcd98b2c656 (127.0.97.2:45341): Not found: Tablet not found: 810502f09dee41118f8a8bdc85d0d560
W20260619 04:37:53.072109 17069 leader_election.cc:341] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7197a25c401474dbcff59d51a5d69a5 (127.0.97.3:43063): Not found: Tablet not found: 810502f09dee41118f8a8bdc85d0d560
I20260619 04:37:53.072209 17069 leader_election.cc:302] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b3ffdf1457645f7a3a566928d3d3c2f; no voters: 1d09e4b187424dfbafc0bbcd98b2c656, e7197a25c401474dbcff59d51a5d69a5
I20260619 04:37:53.072438 17497 raft_consensus.cc:2726] T 810502f09dee41118f8a8bdc85d0d560 P 8b3ffdf1457645f7a3a566928d3d3c2f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 04:37:53.115083 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:53.114282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.2:43045)
0619 04:37:53.114792 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:53.114880 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":241,"threads_started":1}
I20260619 04:37:53.534720 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:53.534001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.3:57309)
0619 04:37:53.534448 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:53.534530 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":216,"threads_started":1}
I20260619 04:37:53.688198 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:53.687502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.1:52547)
0619 04:37:53.687929 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:53.688005 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":225,"threads_started":1}
I20260619 04:37:54.117045 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:54.116307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.2:43841)
0619 04:37:54.116783 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:54.116879 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":255,"threads_started":1}
I20260619 04:37:54.519877 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 17012
I20260619 04:37:54.536725 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:54.535834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43433 (local address 127.0.97.3:45641)
0619 04:37:54.536358 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:54.536490 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":237,"threads_started":1}
I20260619 04:37:54.541136 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:54.520983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59872 (local address 127.0.97.1:39759)
0619 04:37:54.521468 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:54.521478 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:54.521510 (+ 32us) server_negotiation.cc:407] Connection header received
0619 04:37:54.521579 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:54.521585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:54.521841 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:54.521995 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:37:54.523294 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:54.524292 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:54.525495 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:54.526117 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:54.526218 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:54.535135 (+ 8917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:37:54.535157 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:37:54.535178 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:37:54.535230 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:37:54.538387 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:54.539170 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:54.539179 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:54.539181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:54.539329 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:54.539692 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:54.539714 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:54.539716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:54.540087 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:37:54.540322 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:54.540648 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 04:37:54.540887 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":211,"threads_started":1}
I20260619 04:37:54.542404 17178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:37:54.633219 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 17012
I20260619 04:37:54.647904 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 17187
I20260619 04:37:54.661099 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:54.648670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59144 (local address 127.0.97.2:45341)
0619 04:37:54.649164 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:54.649169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:54.649197 (+ 28us) server_negotiation.cc:407] Connection header received
0619 04:37:54.649253 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:54.649257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:54.649318 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:54.649405 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:37:54.650533 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:54.651303 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:54.652512 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:54.652944 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:54.653038 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:54.655463 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:37:54.655468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:37:54.655471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:37:54.655521 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:37:54.658378 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:54.659110 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:54.659115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:54.659117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:54.659237 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:54.659536 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:54.659539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:54.659541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:54.659819 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:37:54.660011 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:54.660486 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 04:37:54.660711 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":250,"threads_started":1}
I20260619 04:37:54.661793 17310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:37:54.744835 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 17187
I20260619 04:37:54.759176 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 17318
I20260619 04:37:54.772254 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:54.759900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45974 (local address 127.0.97.3:43063)
0619 04:37:54.760443 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:54.760469 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:54.760488 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:37:54.760547 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:54.760550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:54.760643 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:54.760742 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:37:54.762058 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:54.762854 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:54.764109 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:54.764568 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:54.764685 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:54.767158 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:37:54.767163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:37:54.767167 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:37:54.767202 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:37:54.769773 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:54.770392 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:54.770397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:54.770401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:54.770530 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:54.770875 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:54.770879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:54.770881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:54.771129 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:37:54.771357 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:54.771695 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 04:37:54.771896 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":252,"threads_started":1}
I20260619 04:37:54.773118 17441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:37:54.854506 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 17318
2026-06-19T04:37:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64061 ms)
[----------] 30 tests from SecurityITest (285931 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:37:54 dist-test-slave-n3tq krb5kdc[17511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:37:54 dist-test-slave-n3tq krb5kdc[17511](info): set up 2 sockets
Jun 19 04:37:54 dist-test-slave-n3tq krb5kdc[17511](info): commencing operation
krb5kdc: starting...
W20260619 04:37:55.868287 388 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:37:56 dist-test-slave-n3tq krb5kdc[17511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:37:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:37:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:37:56.098201 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41691
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:37:56.415663 17524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:37:56.415968 17524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:37:56.416082 17524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:37:56.423337 17524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:37:56.423454 17524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:37:56.423516 17524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:37:56.423592 17524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:37:56.423645 17524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:37:56.433633 17524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:37:56.434959 17524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:37:56.436268 17524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:37:56.447921 17529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:56.447948 17532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:56.447988 17530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:37:56.448731 17524 server_base.cc:1034] running on GCE node
I20260619 04:37:56.449522 17524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:37:56.450927 17524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:37:56.452128 17524 hybrid_clock.cc:648] HybridClock initialized: now 1781843876452079 us; error 37 us; skew 500 ppm
I20260619 04:37:56.452443 17524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:37:56 dist-test-slave-n3tq krb5kdc[17511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:37:56.463049 17524 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:37:56.464746 17524 webserver.cc:460] Webserver started at http://127.0.97.62:42503/ using document root <none> and password file <none>
I20260619 04:37:56.465445 17524 fs_manager.cc:359] Metadata directory not provided
I20260619 04:37:56.465557 17524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:37:56.465860 17524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:37:56.469493 17524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "45e756dfda3244ebab6286cd24c355df"
format_stamp: "Formatted at 2026-06-19 04:37:56 on dist-test-slave-n3tq"
server_key: "0357b822a6ddf759f1cf9f49e09d7b97"
server_key_iv: "b11146b1b692adff204121f733b011e8"
server_key_version: "encryptionkey@0"
I20260619 04:37:56.470494 17524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "45e756dfda3244ebab6286cd24c355df"
format_stamp: "Formatted at 2026-06-19 04:37:56 on dist-test-slave-n3tq"
server_key: "0357b822a6ddf759f1cf9f49e09d7b97"
server_key_iv: "b11146b1b692adff204121f733b011e8"
server_key_version: "encryptionkey@0"
I20260619 04:37:56.477566 17524 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260619 04:37:56.482352 17539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:56.484133 17524 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 04:37:56.484338 17524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "45e756dfda3244ebab6286cd24c355df"
format_stamp: "Formatted at 2026-06-19 04:37:56 on dist-test-slave-n3tq"
server_key: "0357b822a6ddf759f1cf9f49e09d7b97"
server_key_iv: "b11146b1b692adff204121f733b011e8"
server_key_version: "encryptionkey@0"
I20260619 04:37:56.484620 17524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:37:56.502027 17524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:37:56.507601 17524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:37:56.507927 17524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:37:56.524793 17524 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41691
I20260619 04:37:56.524816 17590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41691 every 8 connection(s)
I20260619 04:37:56.526227 17524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:37:56.530121 17591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:37:56.530170 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 17524
I20260619 04:37:56.530345 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:37:56.530999 388 external_mini_cluster.cc:1428] Setting key 297d92088cf7dd73dbe5b563cab751bd
I20260619 04:37:56.544003 17591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df: Bootstrap starting.
Jun 19 04:37:56 dist-test-slave-n3tq krb5kdc[17511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:37:56.547984 17591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df: Neither blocks nor log segments found. Creating new log.
I20260619 04:37:56.549341 17591 log.cc:824] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df: Log is configured to *not* fsync() on all Append() calls
I20260619 04:37:56.552671 17591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df: No bootstrap required, opened a new log
I20260619 04:37:56.555670 17594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:56.532965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34172 (local address 127.0.97.62:41691)
0619 04:37:56.533765 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:56.533776 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:56.533820 (+ 44us) server_negotiation.cc:407] Connection header received
0619 04:37:56.536041 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:56.536069 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:56.536570 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:56.536930 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:37:56.538801 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:56.540123 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:56.542054 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:56.542876 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:56.543029 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:56.547786 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:37:56.547795 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:37:56.547811 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:37:56.547853 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:37:56.551095 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:56.552008 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:56.552015 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:56.552022 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:56.552205 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:56.552708 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:56.552717 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:56.552720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:56.553342 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:37:56.553612 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:56.554129 (+ 517us) server_negotiation.cc:299] Negotiation successful
0619 04:37:56.554525 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":352,"threads_started":1}
I20260619 04:37:56.560984 17591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e756dfda3244ebab6286cd24c355df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41691 } }
I20260619 04:37:56.561383 17591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:37:56.561489 17591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e756dfda3244ebab6286cd24c355df, State: Initialized, Role: FOLLOWER
I20260619 04:37:56.562076 17591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e756dfda3244ebab6286cd24c355df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41691 } }
I20260619 04:37:56.562305 17591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:37:56.562438 17591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:37:56.562587 17591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:37:56.564867 17591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e756dfda3244ebab6286cd24c355df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41691 } }
I20260619 04:37:56.565353 17591 leader_election.cc:302] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45e756dfda3244ebab6286cd24c355df; no voters:
I20260619 04:37:56.565968 17591 leader_election.cc:288] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:37:56.566087 17596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:37:56.566761 17596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df [term 1 LEADER]: Becoming Leader. State: Replica: 45e756dfda3244ebab6286cd24c355df, State: Running, Role: LEADER
I20260619 04:37:56.567286 17596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e756dfda3244ebab6286cd24c355df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41691 } }
I20260619 04:37:56.568005 17591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:37:56.569793 17597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45e756dfda3244ebab6286cd24c355df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e756dfda3244ebab6286cd24c355df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41691 } } }
I20260619 04:37:56.570175 17597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df [sys.catalog]: This master's current role is: LEADER
I20260619 04:37:56.570427 17598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45e756dfda3244ebab6286cd24c355df. Latest consensus state: current_term: 1 leader_uuid: "45e756dfda3244ebab6286cd24c355df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e756dfda3244ebab6286cd24c355df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41691 } } }
I20260619 04:37:56.570719 17598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df [sys.catalog]: This master's current role is: LEADER
I20260619 04:37:56.572996 17601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:37:56.577466 17601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:37:56.586509 17601 catalog_manager.cc:1347] Generated new cluster ID: 180f10dbc6384610ad00d408b53c3073
I20260619 04:37:56.586637 17601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:37:56.602470 17601 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:37:56.604404 17601 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:37:56.615072 17601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df: Generated new TSK 0
I20260619 04:37:56.615942 17601 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 04:37:56.674084 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41691
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:37:56.993017 17617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:37:56.993288 17617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:37:56.993424 17617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:37:57.000656 17617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:37:57.000774 17617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:37:57.000924 17617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:37:57.010711 17617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:37:57.011977 17617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:37:57.013304 17617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:37:57.025774 17622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:57.025866 17625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:57.025884 17623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:37:57.026405 17617 server_base.cc:1034] running on GCE node
I20260619 04:37:57.027020 17617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:37:57.027940 17617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:37:57.029139 17617 hybrid_clock.cc:648] HybridClock initialized: now 1781843877029082 us; error 49 us; skew 500 ppm
I20260619 04:37:57.029433 17617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:37:57 dist-test-slave-n3tq krb5kdc[17511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:37:57.032399 17617 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:37:57.033883 17617 webserver.cc:460] Webserver started at http://127.0.97.1:35171/ using document root <none> and password file <none>
I20260619 04:37:57.034571 17617 fs_manager.cc:359] Metadata directory not provided
I20260619 04:37:57.034691 17617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:37:57.034994 17617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:37:57.037935 17617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "42705251c44c40b298dc977f9464e429"
format_stamp: "Formatted at 2026-06-19 04:37:57 on dist-test-slave-n3tq"
server_key: "b47035bbf358e69955324ecb74637a00"
server_key_iv: "80dbd3b2b152e2054654e17a71cccba9"
server_key_version: "encryptionkey@0"
I20260619 04:37:57.038691 17617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "42705251c44c40b298dc977f9464e429"
format_stamp: "Formatted at 2026-06-19 04:37:57 on dist-test-slave-n3tq"
server_key: "b47035bbf358e69955324ecb74637a00"
server_key_iv: "80dbd3b2b152e2054654e17a71cccba9"
server_key_version: "encryptionkey@0"
I20260619 04:37:57.043813 17617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:37:57.047555 17632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:57.049010 17617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:37:57.049180 17617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "42705251c44c40b298dc977f9464e429"
format_stamp: "Formatted at 2026-06-19 04:37:57 on dist-test-slave-n3tq"
server_key: "b47035bbf358e69955324ecb74637a00"
server_key_iv: "80dbd3b2b152e2054654e17a71cccba9"
server_key_version: "encryptionkey@0"
I20260619 04:37:57.049373 17617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:37:57.075984 17617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:37:57.080420 17617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:37:57.080749 17617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:37:57.081735 17617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:37:57.083073 17617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:37:57.083164 17617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:57.083302 17617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:37:57.083357 17617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:57.114104 17617 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43863
I20260619 04:37:57.114104 17744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43863 every 8 connection(s)
I20260619 04:37:57.115491 17617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:37:57.119192 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 17617
I20260619 04:37:57.119393 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:37:57.120069 388 external_mini_cluster.cc:1428] Setting key 9e5a1f91d972ccb37f1864e15e49502a
Jun 19 04:37:57 dist-test-slave-n3tq krb5kdc[17511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:37:57.135629 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:57.118031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34547 (local address 127.0.97.62:41691)
0619 04:37:57.118471 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:57.118476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:57.120522 (+ 2046us) server_negotiation.cc:407] Connection header received
0619 04:37:57.121507 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:57.121511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:57.121580 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:57.121670 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:37:57.123352 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:57.124315 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:57.125966 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:57.126378 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:57.126508 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:57.130121 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:37:57.130126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:37:57.130130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:37:57.130164 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:37:57.132610 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:57.133355 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:57.133359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:57.133362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:57.133488 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:57.133902 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:57.133906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:57.133908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:57.134178 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:37:57.134310 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:57.135021 (+ 711us) server_negotiation.cc:299] Negotiation successful
0619 04:37:57.135255 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":209,"threads_started":1}
I20260619 04:37:57.136691 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:57.118484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41691 (local address 127.0.97.1:34547)
0619 04:37:57.119203 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:57.119241 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 04:37:57.121227 (+ 1986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:37:57.121966 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:37:57.121978 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:37:57.122553 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:37:57.123168 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:57.123190 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:57.124495 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:57.124499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:57.125816 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:57.125828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:57.126617 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:57.126624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:57.126848 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:57.127511 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:37:57.127534 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:37:57.129877 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:37:57.132750 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:37:57.132760 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:37:57.132778 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:37:57.133214 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:37:57.133593 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:37:57.133598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:37:57.133601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:37:57.133791 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:37:57.134308 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:37:57.134317 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:37:57.134653 (+ 336us) client_negotiation.cc:769] Sending connection context
0619 04:37:57.134894 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 04:37:57.135472 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":293,"threads_started":1}
I20260619 04:37:57.138371 17745 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41691
I20260619 04:37:57.138659 17745 heartbeater.cc:464] Registering TS with master...
I20260619 04:37:57.139357 17745 heartbeater.cc:511] Master 127.0.97.62:41691 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 04:37:57.141752 17556 ts_manager.cc:194] Registered new tserver with Master: 42705251c44c40b298dc977f9464e429 (127.0.97.1:43863)
I20260619 04:37:57.143322 17556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 04:37:57.174115 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41691
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:37:57.495110 17751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:37:57.495394 17751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:37:57.495518 17751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:37:57.502718 17751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:37:57.502840 17751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:37:57.502990 17751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:37:57.512777 17751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:37:57.514074 17751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:37:57.515388 17751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:37:57.528307 17757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:57.528630 17756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:57.528287 17759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:37:57.529548 17751 server_base.cc:1034] running on GCE node
I20260619 04:37:57.530258 17751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:37:57.531387 17751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:37:57.532606 17751 hybrid_clock.cc:648] HybridClock initialized: now 1781843877532547 us; error 43 us; skew 500 ppm
I20260619 04:37:57.532990 17751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:37:57 dist-test-slave-n3tq krb5kdc[17511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:37:57.536755 17751 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:37:57.538671 17751 webserver.cc:460] Webserver started at http://127.0.97.2:39295/ using document root <none> and password file <none>
I20260619 04:37:57.539548 17751 fs_manager.cc:359] Metadata directory not provided
I20260619 04:37:57.539676 17751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:37:57.539994 17751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:37:57.543787 17751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "4f49c3e81be146fb925824f0ac2eb50e"
format_stamp: "Formatted at 2026-06-19 04:37:57 on dist-test-slave-n3tq"
server_key: "41d954fb1b08d3c25bd50e7e662b1148"
server_key_iv: "8e1afa75d42aca0187b92b04eebaa826"
server_key_version: "encryptionkey@0"
I20260619 04:37:57.544593 17751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4f49c3e81be146fb925824f0ac2eb50e"
format_stamp: "Formatted at 2026-06-19 04:37:57 on dist-test-slave-n3tq"
server_key: "41d954fb1b08d3c25bd50e7e662b1148"
server_key_iv: "8e1afa75d42aca0187b92b04eebaa826"
server_key_version: "encryptionkey@0"
I20260619 04:37:57.549551 17751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 04:37:57.552882 17766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:57.554243 17751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:37:57.554376 17751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "4f49c3e81be146fb925824f0ac2eb50e"
format_stamp: "Formatted at 2026-06-19 04:37:57 on dist-test-slave-n3tq"
server_key: "41d954fb1b08d3c25bd50e7e662b1148"
server_key_iv: "8e1afa75d42aca0187b92b04eebaa826"
server_key_version: "encryptionkey@0"
I20260619 04:37:57.554541 17751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:37:57.580880 17751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:37:57.584878 17751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:37:57.585117 17751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:37:57.586066 17751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:37:57.587411 17751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:37:57.587482 17751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:57.587570 17751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:37:57.587611 17751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:57.617262 17751 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36989
I20260619 04:37:57.617280 17878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36989 every 8 connection(s)
I20260619 04:37:57.618480 17751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:37:57.626864 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 17751
I20260619 04:37:57.627035 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:37:57.627527 388 external_mini_cluster.cc:1428] Setting key 6bf37ed13122f9e871ff24544c013b62
Jun 19 04:37:57 dist-test-slave-n3tq krb5kdc[17511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:37:57.638592 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:57.620931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47753 (local address 127.0.97.62:41691)
0619 04:37:57.621092 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:57.621099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:57.622169 (+ 1070us) server_negotiation.cc:407] Connection header received
0619 04:37:57.624030 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:57.624036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:57.624120 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:57.624222 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:37:57.625794 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:57.626499 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:57.628121 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:57.628703 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:57.628822 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:57.632499 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:37:57.632506 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:37:57.632510 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:37:57.632548 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:37:57.635546 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:57.636276 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:57.636282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:57.636285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:57.636433 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:57.636826 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:57.636829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:57.636833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:57.637143 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:37:57.637274 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:57.638160 (+ 886us) server_negotiation.cc:299] Negotiation successful
0619 04:37:57.638353 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 04:37:57.640100 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:57.621333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41691 (local address 127.0.97.2:47753)
0619 04:37:57.622003 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:57.622045 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 04:37:57.623754 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:37:57.624495 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:37:57.624508 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:37:57.625032 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:37:57.625615 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:57.625633 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:57.626650 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:57.626654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:57.627963 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:57.627981 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:57.628964 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:57.628971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:57.629270 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:57.630045 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:37:57.630068 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:37:57.632229 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:37:57.635693 (+ 3464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:37:57.635700 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:37:57.635715 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:37:57.636163 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:37:57.636551 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:37:57.636555 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:37:57.636557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:37:57.636736 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:37:57.637313 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:37:57.637322 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:37:57.637741 (+ 419us) client_negotiation.cc:769] Sending connection context
0619 04:37:57.638068 (+ 327us) client_negotiation.cc:240] Negotiation successful
0619 04:37:57.638665 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":267,"threads_started":1}
I20260619 04:37:57.641609 17879 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41691
I20260619 04:37:57.641906 17879 heartbeater.cc:464] Registering TS with master...
I20260619 04:37:57.642580 17879 heartbeater.cc:511] Master 127.0.97.62:41691 requested a full tablet report, sending...
I20260619 04:37:57.644062 17556 ts_manager.cc:194] Registered new tserver with Master: 4f49c3e81be146fb925824f0ac2eb50e (127.0.97.2:36989)
I20260619 04:37:57.645182 17556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47753
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 04:37:57.679311 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41691
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:37:57.998855 17884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:37:57.999131 17884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:37:57.999223 17884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:37:58.006480 17884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:37:58.006595 17884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:37:58.006703 17884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:37:58.016392 17884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:37:58.017673 17884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:37:58.018949 17884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:37:58.031515 17889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:58.031486 17890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:58.031452 17892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:37:58.032121 17884 server_base.cc:1034] running on GCE node
I20260619 04:37:58.032764 17884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:37:58.033622 17884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:37:58.034823 17884 hybrid_clock.cc:648] HybridClock initialized: now 1781843878034780 us; error 34 us; skew 500 ppm
I20260619 04:37:58.035152 17884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:37:58 dist-test-slave-n3tq krb5kdc[17511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:37:58.038122 17884 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:37:58.039480 17884 webserver.cc:460] Webserver started at http://127.0.97.3:40211/ using document root <none> and password file <none>
I20260619 04:37:58.040126 17884 fs_manager.cc:359] Metadata directory not provided
I20260619 04:37:58.040226 17884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:37:58.040531 17884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:37:58.043323 17884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "7a76a45d64584f6e9ff5e4a0c134d698"
format_stamp: "Formatted at 2026-06-19 04:37:58 on dist-test-slave-n3tq"
server_key: "f24cc41b7d5b2fa8e6ba081b0a4d8354"
server_key_iv: "1adfe20ecd338502602f4985e9ffe8dc"
server_key_version: "encryptionkey@0"
I20260619 04:37:58.044062 17884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7a76a45d64584f6e9ff5e4a0c134d698"
format_stamp: "Formatted at 2026-06-19 04:37:58 on dist-test-slave-n3tq"
server_key: "f24cc41b7d5b2fa8e6ba081b0a4d8354"
server_key_iv: "1adfe20ecd338502602f4985e9ffe8dc"
server_key_version: "encryptionkey@0"
I20260619 04:37:58.049192 17884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 04:37:58.052652 17899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:58.053993 17884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:37:58.054136 17884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "7a76a45d64584f6e9ff5e4a0c134d698"
format_stamp: "Formatted at 2026-06-19 04:37:58 on dist-test-slave-n3tq"
server_key: "f24cc41b7d5b2fa8e6ba081b0a4d8354"
server_key_iv: "1adfe20ecd338502602f4985e9ffe8dc"
server_key_version: "encryptionkey@0"
I20260619 04:37:58.054347 17884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:37:58.066608 17884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:37:58.070875 17884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:37:58.071141 17884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:37:58.072111 17884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:37:58.073468 17884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:37:58.073546 17884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:58.073711 17884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:37:58.073773 17884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:58.103161 17884 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43047
I20260619 04:37:58.103206 18011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43047 every 8 connection(s)
I20260619 04:37:58.104404 17884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:37:58.111653 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 17884
I20260619 04:37:58.111814 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:37:58.112223 388 external_mini_cluster.cc:1428] Setting key d866ee3157710582cc9022312067a97e
Jun 19 04:37:58 dist-test-slave-n3tq krb5kdc[17511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:37:58.125599 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:58.107056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46779 (local address 127.0.97.62:41691)
0619 04:37:58.107251 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:58.107256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:58.108371 (+ 1115us) server_negotiation.cc:407] Connection header received
0619 04:37:58.110261 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:58.110267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:58.110354 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:58.110440 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:37:58.112008 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.112894 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.114896 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.115398 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.115527 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:58.119822 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:37:58.119828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:37:58.119832 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:37:58.119906 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:37:58.122614 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:58.123402 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:58.123407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:58.123411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:58.123565 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:58.124023 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:58.124026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:58.124029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:58.124292 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:37:58.124407 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:58.125162 (+ 755us) server_negotiation.cc:299] Negotiation successful
0619 04:37:58.125340 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 04:37:58.126756 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:58.107523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41691 (local address 127.0.97.3:46779)
0619 04:37:58.108210 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:58.108251 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 04:37:58.109976 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:37:58.110687 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:37:58.110696 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:37:58.111223 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:37:58.111835 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:58.111857 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.113048 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.113052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:58.114758 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:58.114772 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.115664 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.115673 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:58.115973 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:58.116746 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:37:58.116777 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:37:58.119539 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:37:58.122775 (+ 3236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:37:58.122783 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:37:58.122796 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:37:58.123241 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:37:58.123729 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:37:58.123733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:37:58.123735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:37:58.123929 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:37:58.124421 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:37:58.124428 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:37:58.124812 (+ 384us) client_negotiation.cc:769] Sending connection context
0619 04:37:58.125047 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 04:37:58.125466 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":262,"threads_started":1}
I20260619 04:37:58.128182 18012 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41691
I20260619 04:37:58.128531 18012 heartbeater.cc:464] Registering TS with master...
I20260619 04:37:58.129238 18012 heartbeater.cc:511] Master 127.0.97.62:41691 requested a full tablet report, sending...
I20260619 04:37:58.130653 17556 ts_manager.cc:194] Registered new tserver with Master: 7a76a45d64584f6e9ff5e4a0c134d698 (127.0.97.3:43047)
I20260619 04:37:58.131471 17556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46779
I20260619 04:37:58.140378 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:37:58.146669 17745 heartbeater.cc:503] Master 127.0.97.62:41691 was elected leader, sending a full tablet report...
I20260619 04:37:58.157056 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:58.144836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43358 (local address 127.0.97.62:41691)
0619 04:37:58.144994 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:58.144999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:58.145461 (+ 462us) server_negotiation.cc:407] Connection header received
0619 04:37:58.145574 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:58.145577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:58.145656 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:58.145741 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:37:58.147222 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.148004 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.149468 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.149856 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.149957 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:58.152158 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:37:58.152163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:37:58.152167 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:37:58.152197 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:37:58.154733 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:58.155336 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:58.155340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:58.155342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:58.155452 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:58.155735 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:58.155738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:58.155740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:58.156037 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:37:58.156228 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:58.156649 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 04:37:58.156830 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 04:37:58.161007 17556 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:37:58.168274 17556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:37:58.190969 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:58.180419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43863 (local address 127.0.0.1:38136)
0619 04:37:58.181206 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:58.181258 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 04:37:58.181445 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:37:58.182733 (+ 1288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:37:58.182740 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:37:58.182772 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:37:58.183172 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:58.183187 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.185423 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.185429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:58.187783 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:58.187801 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.189871 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.189879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:58.190109 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:58.190197 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 04:37:58.190324 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 04:37:58.190543 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":713,"threads_started":1}
I20260619 04:37:58.192613 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:58.180793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38136 (local address 127.0.97.1:43863)
0619 04:37:58.182041 (+ 1248us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:58.182049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:58.182079 (+ 30us) server_negotiation.cc:407] Connection header received
0619 04:37:58.182176 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:58.182184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:58.182498 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:58.183391 (+ 893us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:37:58.183609 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.185273 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.188020 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.189620 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.191700 (+ 2080us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:58.191978 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:58.192056 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 04:37:58.192247 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1094,"thread_start_us":966,"threads_started":1}
I20260619 04:37:58.198962 17680 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7c2d4ccf1c4eb8bf05793dbbb59d4a (DEFAULT_TABLE table=test-table [id=70d949713acb4f219ee39c13f1c35234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:37:58.200479 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:58.180595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36989 (local address 127.0.0.1:40470)
0619 04:37:58.191258 (+ 10663us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:58.191280 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 04:37:58.191391 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:37:58.193733 (+ 2342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:37:58.193750 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:37:58.193786 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:37:58.193977 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:58.193988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.196157 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.196162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:58.198000 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:58.198014 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.199697 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.199702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:58.199895 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:58.199931 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 04:37:58.200025 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 04:37:58.200168 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10544,"thread_start_us":16433,"threads_started":1}
I20260619 04:37:58.200563 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:58.181045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40470 (local address 127.0.97.2:36989)
0619 04:37:58.193002 (+ 11957us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:58.193012 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:58.193046 (+ 34us) server_negotiation.cc:407] Connection header received
0619 04:37:58.193158 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:58.193168 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:58.193469 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:58.194173 (+ 704us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:37:58.194400 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.195985 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.198176 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.199542 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.199670 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:58.199876 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:58.200081 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 04:37:58.200247 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11783,"thread_start_us":11439,"threads_started":1}
I20260619 04:37:58.200925 17680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7c2d4ccf1c4eb8bf05793dbbb59d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:37:58.201623 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:58.182594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43047 (local address 127.0.0.1:40120)
0619 04:37:58.193845 (+ 11251us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:58.193863 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:37:58.193985 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:37:58.194946 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:37:58.194951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:37:58.194976 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:37:58.195366 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:58.195377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.196981 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.196986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:58.199200 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:58.199213 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.200912 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.200918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:58.201080 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:58.201111 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 04:37:58.201202 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 04:37:58.201328 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11107,"thread_start_us":10963,"threads_started":1}
I20260619 04:37:58.202402 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:58.183392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40120 (local address 127.0.97.3:43047)
0619 04:37:58.193387 (+ 9995us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:58.193397 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:58.194108 (+ 711us) server_negotiation.cc:407] Connection header received
0619 04:37:58.194234 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:58.194241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:58.194499 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:58.194684 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:37:58.195573 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.196848 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.199353 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.200659 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.200782 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:58.200994 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:58.201271 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 04:37:58.201424 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9823,"thread_start_us":9644,"threads_started":1}
I20260619 04:37:58.204260 17814 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7c2d4ccf1c4eb8bf05793dbbb59d4a (DEFAULT_TABLE table=test-table [id=70d949713acb4f219ee39c13f1c35234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:37:58.205781 17814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7c2d4ccf1c4eb8bf05793dbbb59d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:37:58.206416 17947 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7c2d4ccf1c4eb8bf05793dbbb59d4a (DEFAULT_TABLE table=test-table [id=70d949713acb4f219ee39c13f1c35234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:37:58.208022 17947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7c2d4ccf1c4eb8bf05793dbbb59d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:37:58.212788 18030 tablet_bootstrap.cc:492] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 42705251c44c40b298dc977f9464e429: Bootstrap starting.
I20260619 04:37:58.216387 18030 tablet_bootstrap.cc:654] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 42705251c44c40b298dc977f9464e429: Neither blocks nor log segments found. Creating new log.
I20260619 04:37:58.217403 18030 log.cc:824] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 42705251c44c40b298dc977f9464e429: Log is configured to *not* fsync() on all Append() calls
I20260619 04:37:58.219956 18030 tablet_bootstrap.cc:492] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 42705251c44c40b298dc977f9464e429: No bootstrap required, opened a new log
I20260619 04:37:58.220340 18030 ts_tablet_manager.cc:1397] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 42705251c44c40b298dc977f9464e429: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 04:37:58.220472 18032 tablet_bootstrap.cc:492] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 4f49c3e81be146fb925824f0ac2eb50e: Bootstrap starting.
I20260619 04:37:58.222380 18033 tablet_bootstrap.cc:492] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698: Bootstrap starting.
I20260619 04:37:58.225179 18032 tablet_bootstrap.cc:654] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 4f49c3e81be146fb925824f0ac2eb50e: Neither blocks nor log segments found. Creating new log.
I20260619 04:37:58.225203 18033 tablet_bootstrap.cc:654] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698: Neither blocks nor log segments found. Creating new log.
I20260619 04:37:58.226409 18033 log.cc:824] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698: Log is configured to *not* fsync() on all Append() calls
I20260619 04:37:58.226423 18032 log.cc:824] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 4f49c3e81be146fb925824f0ac2eb50e: Log is configured to *not* fsync() on all Append() calls
I20260619 04:37:58.229244 18032 tablet_bootstrap.cc:492] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 4f49c3e81be146fb925824f0ac2eb50e: No bootstrap required, opened a new log
I20260619 04:37:58.229470 18033 tablet_bootstrap.cc:492] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698: No bootstrap required, opened a new log
I20260619 04:37:58.229504 18032 ts_tablet_manager.cc:1397] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 4f49c3e81be146fb925824f0ac2eb50e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 04:37:58.229722 18033 ts_tablet_manager.cc:1397] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 04:37:58.230785 18030 raft_consensus.cc:348] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 42705251c44c40b298dc977f9464e429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42705251c44c40b298dc977f9464e429" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } } peers { permanent_uuid: "4f49c3e81be146fb925824f0ac2eb50e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36989 } } peers { permanent_uuid: "7a76a45d64584f6e9ff5e4a0c134d698" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43047 } }
I20260619 04:37:58.231247 18030 raft_consensus.cc:374] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 42705251c44c40b298dc977f9464e429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:37:58.231388 18030 raft_consensus.cc:729] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 42705251c44c40b298dc977f9464e429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42705251c44c40b298dc977f9464e429, State: Initialized, Role: FOLLOWER
I20260619 04:37:58.232201 18030 consensus_queue.cc:260] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 42705251c44c40b298dc977f9464e429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42705251c44c40b298dc977f9464e429" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } } peers { permanent_uuid: "4f49c3e81be146fb925824f0ac2eb50e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36989 } } peers { permanent_uuid: "7a76a45d64584f6e9ff5e4a0c134d698" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43047 } }
I20260619 04:37:58.234006 18030 ts_tablet_manager.cc:1428] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 42705251c44c40b298dc977f9464e429: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260619 04:37:58.237522 18032 raft_consensus.cc:348] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 4f49c3e81be146fb925824f0ac2eb50e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42705251c44c40b298dc977f9464e429" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } } peers { permanent_uuid: "4f49c3e81be146fb925824f0ac2eb50e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36989 } } peers { permanent_uuid: "7a76a45d64584f6e9ff5e4a0c134d698" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43047 } }
I20260619 04:37:58.238008 18032 raft_consensus.cc:374] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 4f49c3e81be146fb925824f0ac2eb50e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:37:58.238117 18032 raft_consensus.cc:729] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 4f49c3e81be146fb925824f0ac2eb50e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f49c3e81be146fb925824f0ac2eb50e, State: Initialized, Role: FOLLOWER
I20260619 04:37:58.238785 18032 consensus_queue.cc:260] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 4f49c3e81be146fb925824f0ac2eb50e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42705251c44c40b298dc977f9464e429" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } } peers { permanent_uuid: "4f49c3e81be146fb925824f0ac2eb50e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36989 } } peers { permanent_uuid: "7a76a45d64584f6e9ff5e4a0c134d698" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43047 } }
I20260619 04:37:58.239933 17879 heartbeater.cc:503] Master 127.0.97.62:41691 was elected leader, sending a full tablet report...
I20260619 04:37:58.239897 18033 raft_consensus.cc:348] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42705251c44c40b298dc977f9464e429" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } } peers { permanent_uuid: "4f49c3e81be146fb925824f0ac2eb50e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36989 } } peers { permanent_uuid: "7a76a45d64584f6e9ff5e4a0c134d698" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43047 } }
I20260619 04:37:58.240262 18033 raft_consensus.cc:374] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:37:58.240346 18033 raft_consensus.cc:729] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a76a45d64584f6e9ff5e4a0c134d698, State: Initialized, Role: FOLLOWER
I20260619 04:37:58.241025 18033 consensus_queue.cc:260] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42705251c44c40b298dc977f9464e429" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } } peers { permanent_uuid: "4f49c3e81be146fb925824f0ac2eb50e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36989 } } peers { permanent_uuid: "7a76a45d64584f6e9ff5e4a0c134d698" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43047 } }
I20260619 04:37:58.241372 18032 ts_tablet_manager.cc:1428] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 4f49c3e81be146fb925824f0ac2eb50e: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260619 04:37:58.242669 18012 heartbeater.cc:503] Master 127.0.97.62:41691 was elected leader, sending a full tablet report...
I20260619 04:37:58.243271 18033 ts_tablet_manager.cc:1428] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 04:37:58.345494 18038 raft_consensus.cc:482] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:37:58.345804 18038 raft_consensus.cc:504] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42705251c44c40b298dc977f9464e429" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } } peers { permanent_uuid: "4f49c3e81be146fb925824f0ac2eb50e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36989 } } peers { permanent_uuid: "7a76a45d64584f6e9ff5e4a0c134d698" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43047 } }
I20260619 04:37:58.347604 18038 leader_election.cc:288] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42705251c44c40b298dc977f9464e429 (127.0.97.1:43863), 4f49c3e81be146fb925824f0ac2eb50e (127.0.97.2:36989)
I20260619 04:37:58.353750 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:58.348116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43087 (local address 127.0.97.1:43863)
0619 04:37:58.348349 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:58.348355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:58.348374 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:37:58.348479 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:58.348484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:58.348565 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:58.348650 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:37:58.349137 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.350071 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.351845 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.352948 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.353056 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:58.353147 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:58.353444 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 04:37:58.353536 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 04:37:58.353896 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:58.347792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43863 (local address 127.0.97.3:43087)
0619 04:37:58.348089 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:58.348108 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:37:58.348247 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:37:58.348759 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:37:58.348764 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:37:58.348799 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:37:58.348988 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:58.349001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.350212 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.350218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:58.351683 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:58.351695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.353093 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.353100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:58.353286 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:58.353321 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 04:37:58.353428 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 04:37:58.353560 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":22}
I20260619 04:37:58.354584 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:58.347890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36989 (local address 127.0.97.3:58547)
0619 04:37:58.348850 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0619 04:37:58.348867 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:37:58.348989 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:37:58.349287 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:37:58.349292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:37:58.349315 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:37:58.349650 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:58.349660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.350804 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.350808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:58.352404 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:37:58.352419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.353929 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.353935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:37:58.354097 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:58.354127 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 04:37:58.354210 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 04:37:58.354296 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":1,"thread_start_us":620,"threads_started":1}
I20260619 04:37:58.354611 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:58.348502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58547 (local address 127.0.97.2:36989)
0619 04:37:58.348710 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:58.348716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:58.348963 (+ 247us) server_negotiation.cc:407] Connection header received
0619 04:37:58.349118 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:58.349123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:58.349184 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:58.349258 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:37:58.349783 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.350637 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.352574 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.353558 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.353690 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:58.353781 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:58.354246 (+ 465us) server_negotiation.cc:299] Negotiation successful
0619 04:37:58.354372 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 04:37:58.355018 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7c2d4ccf1c4eb8bf05793dbbb59d4a" candidate_uuid: "7a76a45d64584f6e9ff5e4a0c134d698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42705251c44c40b298dc977f9464e429" is_pre_election: true
I20260619 04:37:58.355590 17700 raft_consensus.cc:2454] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 42705251c44c40b298dc977f9464e429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a76a45d64584f6e9ff5e4a0c134d698 in term 0.
I20260619 04:37:58.355589 17834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7c2d4ccf1c4eb8bf05793dbbb59d4a" candidate_uuid: "7a76a45d64584f6e9ff5e4a0c134d698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f49c3e81be146fb925824f0ac2eb50e" is_pre_election: true
W20260619 04:37:58.355984 18013 tablet.cc:2369] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:37:58.356101 17834 raft_consensus.cc:2454] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 4f49c3e81be146fb925824f0ac2eb50e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a76a45d64584f6e9ff5e4a0c134d698 in term 0.
I20260619 04:37:58.356316 17901 leader_election.cc:302] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42705251c44c40b298dc977f9464e429, 7a76a45d64584f6e9ff5e4a0c134d698; no voters:
I20260619 04:37:58.356781 18038 raft_consensus.cc:2781] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:37:58.356932 18038 raft_consensus.cc:482] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:37:58.357014 18038 raft_consensus.cc:3037] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:37:58.359319 18038 raft_consensus.cc:504] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42705251c44c40b298dc977f9464e429" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } } peers { permanent_uuid: "4f49c3e81be146fb925824f0ac2eb50e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36989 } } peers { permanent_uuid: "7a76a45d64584f6e9ff5e4a0c134d698" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43047 } }
I20260619 04:37:58.359853 18038 leader_election.cc:288] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698 [CANDIDATE]: Term 1 election: Requested vote from peers 42705251c44c40b298dc977f9464e429 (127.0.97.1:43863), 4f49c3e81be146fb925824f0ac2eb50e (127.0.97.2:36989)
I20260619 04:37:58.360328 17834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7c2d4ccf1c4eb8bf05793dbbb59d4a" candidate_uuid: "7a76a45d64584f6e9ff5e4a0c134d698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f49c3e81be146fb925824f0ac2eb50e"
I20260619 04:37:58.360328 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7c2d4ccf1c4eb8bf05793dbbb59d4a" candidate_uuid: "7a76a45d64584f6e9ff5e4a0c134d698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42705251c44c40b298dc977f9464e429"
I20260619 04:37:58.360560 17834 raft_consensus.cc:3037] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 4f49c3e81be146fb925824f0ac2eb50e [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:37:58.360560 17700 raft_consensus.cc:3037] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 42705251c44c40b298dc977f9464e429 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:37:58.363623 17700 raft_consensus.cc:2454] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 42705251c44c40b298dc977f9464e429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a76a45d64584f6e9ff5e4a0c134d698 in term 1.
I20260619 04:37:58.363629 17834 raft_consensus.cc:2454] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 4f49c3e81be146fb925824f0ac2eb50e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a76a45d64584f6e9ff5e4a0c134d698 in term 1.
I20260619 04:37:58.364049 17901 leader_election.cc:302] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42705251c44c40b298dc977f9464e429, 7a76a45d64584f6e9ff5e4a0c134d698; no voters:
I20260619 04:37:58.364347 18038 raft_consensus.cc:2781] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:37:58.364780 18038 raft_consensus.cc:686] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698 [term 1 LEADER]: Becoming Leader. State: Replica: 7a76a45d64584f6e9ff5e4a0c134d698, State: Running, Role: LEADER
I20260619 04:37:58.365329 18038 consensus_queue.cc:237] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42705251c44c40b298dc977f9464e429" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } } peers { permanent_uuid: "4f49c3e81be146fb925824f0ac2eb50e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36989 } } peers { permanent_uuid: "7a76a45d64584f6e9ff5e4a0c134d698" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43047 } }
W20260619 04:37:58.367940 17746 tablet.cc:2369] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 42705251c44c40b298dc977f9464e429: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:37:58.369808 17556 catalog_manager.cc:5495] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a76a45d64584f6e9ff5e4a0c134d698 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "7a76a45d64584f6e9ff5e4a0c134d698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42705251c44c40b298dc977f9464e429" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f49c3e81be146fb925824f0ac2eb50e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a76a45d64584f6e9ff5e4a0c134d698" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43047 } health_report { overall_health: HEALTHY } } }
W20260619 04:37:58.370252 17880 tablet.cc:2369] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 4f49c3e81be146fb925824f0ac2eb50e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:37:58.397120 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:58.392180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40122 (local address 127.0.97.3:43047)
0619 04:37:58.392353 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:58.392358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:58.392474 (+ 116us) server_negotiation.cc:407] Connection header received
0619 04:37:58.392676 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:58.392680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:58.392758 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:58.392887 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:37:58.393177 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.393918 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.395247 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.395632 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.395718 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:58.396130 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:37:58.396690 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:37:58.396793 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:58.396860 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 04:37:58.396924 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 04:37:58.404986 17834 raft_consensus.cc:1261] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 4f49c3e81be146fb925824f0ac2eb50e [term 1 FOLLOWER]: Refusing update from remote peer 7a76a45d64584f6e9ff5e4a0c134d698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:37:58.404986 17700 raft_consensus.cc:1261] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 42705251c44c40b298dc977f9464e429 [term 1 FOLLOWER]: Refusing update from remote peer 7a76a45d64584f6e9ff5e4a0c134d698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:37:58.405929 18040 consensus_queue.cc:1035] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42705251c44c40b298dc977f9464e429" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:37:58.406260 18038 consensus_queue.cc:1035] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f49c3e81be146fb925824f0ac2eb50e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:37:58.424870 18044 mvcc.cc:205] Tried to move back new op lower bound from 7298432525937606656 to 7298432525788303360. Current Snapshot: MvccSnapshot[applied={T|T < 7298432525937606656}]
I20260619 04:37:58.434324 17555 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:37:58.434675 17555 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:37:58.438445 17555 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45e756dfda3244ebab6286cd24c355df: Sending DeleteTablet for 3 replicas of tablet 3b7c2d4ccf1c4eb8bf05793dbbb59d4a
I20260619 04:37:58.439658 17947 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7c2d4ccf1c4eb8bf05793dbbb59d4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:37:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40120
I20260619 04:37:58.439713 17680 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7c2d4ccf1c4eb8bf05793dbbb59d4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:37:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38136
I20260619 04:37:58.439688 17814 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7c2d4ccf1c4eb8bf05793dbbb59d4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:37:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40470
I20260619 04:37:58.445397 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 17617
I20260619 04:37:58.452113 18054 tablet_replica.cc:330] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 42705251c44c40b298dc977f9464e429: stopping tablet replica
I20260619 04:37:58.452961 18055 tablet_replica.cc:330] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 4f49c3e81be146fb925824f0ac2eb50e: stopping tablet replica
I20260619 04:37:58.453378 18055 raft_consensus.cc:2229] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 4f49c3e81be146fb925824f0ac2eb50e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:37:58.453429 18054 raft_consensus.cc:2229] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 42705251c44c40b298dc977f9464e429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:37:58.453912 18055 raft_consensus.cc:2258] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 4f49c3e81be146fb925824f0ac2eb50e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:37:58.454201 18054 raft_consensus.cc:2258] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 42705251c44c40b298dc977f9464e429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:37:58.455235 18053 tablet_replica.cc:330] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698: stopping tablet replica
I20260619 04:37:58.455760 18053 raft_consensus.cc:2229] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:37:58.456139 18055 ts_tablet_manager.cc:1905] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 4f49c3e81be146fb925824f0ac2eb50e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:37:58.456518 18053 raft_consensus.cc:2258] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:37:58.458823 18053 ts_tablet_manager.cc:1905] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:37:58.458962 18054 ts_tablet_manager.cc:1905] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 42705251c44c40b298dc977f9464e429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:37:58.462881 18055 ts_tablet_manager.cc:1918] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 4f49c3e81be146fb925824f0ac2eb50e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:37:58.463035 18055 log.cc:1196] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 4f49c3e81be146fb925824f0ac2eb50e: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/3b7c2d4ccf1c4eb8bf05793dbbb59d4a
I20260619 04:37:58.463512 18055 ts_tablet_manager.cc:1939] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 4f49c3e81be146fb925824f0ac2eb50e: Deleting consensus metadata
I20260619 04:37:58.465354 17543 catalog_manager.cc:4841] TS 4f49c3e81be146fb925824f0ac2eb50e (127.0.97.2:36989): tablet 3b7c2d4ccf1c4eb8bf05793dbbb59d4a (table test-table [id=70d949713acb4f219ee39c13f1c35234]) successfully deleted
I20260619 04:37:58.465435 18054 ts_tablet_manager.cc:1918] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 42705251c44c40b298dc977f9464e429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:37:58.465574 18054 log.cc:1196] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 42705251c44c40b298dc977f9464e429: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-0/wal/wals/3b7c2d4ccf1c4eb8bf05793dbbb59d4a
I20260619 04:37:58.465996 18054 ts_tablet_manager.cc:1939] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 42705251c44c40b298dc977f9464e429: Deleting consensus metadata
I20260619 04:37:58.466737 18053 ts_tablet_manager.cc:1918] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:37:58.466922 18053 log.cc:1196] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/3b7c2d4ccf1c4eb8bf05793dbbb59d4a
I20260619 04:37:58.467404 18053 ts_tablet_manager.cc:1939] T 3b7c2d4ccf1c4eb8bf05793dbbb59d4a P 7a76a45d64584f6e9ff5e4a0c134d698: Deleting consensus metadata
I20260619 04:37:58.467584 17541 catalog_manager.cc:4841] TS 42705251c44c40b298dc977f9464e429 (127.0.97.1:43863): tablet 3b7c2d4ccf1c4eb8bf05793dbbb59d4a (table test-table [id=70d949713acb4f219ee39c13f1c35234]) successfully deleted
I20260619 04:37:58.469195 17541 catalog_manager.cc:4841] TS 7a76a45d64584f6e9ff5e4a0c134d698 (127.0.97.3:43047): tablet 3b7c2d4ccf1c4eb8bf05793dbbb59d4a (table test-table [id=70d949713acb4f219ee39c13f1c35234]) successfully deleted
Jun 19 04:37:58 dist-test-slave-n3tq krb5kdc[17511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 04:37:58.477515 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:58.447005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38146 (local address 127.0.97.1:43863)
0619 04:37:58.447192 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:58.447199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:58.453205 (+ 6006us) server_negotiation.cc:407] Connection header received
0619 04:37:58.453501 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:58.453507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:58.453585 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:58.453664 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:37:58.455602 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.456643 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.458680 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.459269 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.459365 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:58.471004 (+ 11639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:37:58.471020 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:37:58.471038 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:37:58.471089 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:37:58.474812 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:58.475456 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:58.475465 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:58.475469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:58.475632 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:58.475980 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:58.475984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:58.475987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:58.476433 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:37:58.476754 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:58.476925 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 04:37:58.477143 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:37:58.478163 17740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:37:58.567584 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 17617
I20260619 04:37:58.583034 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 17751
Jun 19 04:37:58 dist-test-slave-n3tq krb5kdc[17511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 04:37:58.601615 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:58.583822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40478 (local address 127.0.97.2:36989)
0619 04:37:58.584200 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:58.584204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:58.584219 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:37:58.584272 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:58.584276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:58.584335 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:58.584417 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:37:58.586130 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.586848 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.588822 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.589273 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.589382 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:58.594903 (+ 5521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:37:58.594927 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:37:58.594948 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:37:58.595009 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:37:58.598678 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:58.599476 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:58.599488 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:58.599492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:58.599654 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:58.599987 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:58.599992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:58.599996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:58.600519 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:37:58.600817 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:58.601008 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 04:37:58.601233 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268}
I20260619 04:37:58.602283 17874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:37:58.695362 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 17751
I20260619 04:37:58.716043 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 17884
Jun 19 04:37:58 dist-test-slave-n3tq krb5kdc[17511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 04:37:58.734606 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:58.716780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40132 (local address 127.0.97.3:43047)
0619 04:37:58.716964 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:58.716969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:58.716987 (+ 18us) server_negotiation.cc:407] Connection header received
0619 04:37:58.717183 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:58.717187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:58.717247 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:58.717349 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:37:58.719095 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.719927 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.721948 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:58.722412 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:58.722522 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:58.727829 (+ 5307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:37:58.727848 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:37:58.727869 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:37:58.727923 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:37:58.731750 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:58.732544 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:58.732558 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:58.732561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:58.732721 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:37:58.733139 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:37:58.733142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:37:58.733144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:37:58.733513 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:37:58.733752 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:58.734106 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 04:37:58.734270 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 04:37:58.735378 18007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:37:58.857728 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 17884
I20260619 04:37:58.873534 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 17524
I20260619 04:37:58.874735 17586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:37:58.984373 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 17524
2026-06-19T04:37:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4132 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T04:37:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:37:59Z Disabled control of system clock
I20260619 04:37:59.054195 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45057
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 04:37:59.378508 18064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:37:59.378794 18064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:37:59.378878 18064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:37:59.386111 18064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:37:59.386227 18064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:37:59.386263 18064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:37:59.386315 18064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:37:59.386356 18064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:37:59.395578 18064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45057
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:37:59.396847 18064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:37:59.398662 18064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:37:59.410032 18070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:59.410044 18072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:59.410334 18069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:37:59.410868 18064 server_base.cc:1034] running on GCE node
I20260619 04:37:59.411682 18064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:37:59.413182 18064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:37:59.414389 18064 hybrid_clock.cc:648] HybridClock initialized: now 1781843879414343 us; error 40 us; skew 500 ppm
I20260619 04:37:59.414722 18064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:37:59.417017 18064 webserver.cc:460] Webserver started at http://127.0.97.62:43041/ using document root <none> and password file <none>
I20260619 04:37:59.417717 18064 fs_manager.cc:359] Metadata directory not provided
I20260619 04:37:59.417817 18064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:37:59.418136 18064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:37:59.421145 18064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "8647a69f71cb41daa622e72060904cbf"
format_stamp: "Formatted at 2026-06-19 04:37:59 on dist-test-slave-n3tq"
server_key: "6df3ffa3ba895b0cda3c03b6f6d6a9a7"
server_key_iv: "bacbf697c06340a1a680fc742e837c93"
server_key_version: "encryptionkey@0"
I20260619 04:37:59.421922 18064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "8647a69f71cb41daa622e72060904cbf"
format_stamp: "Formatted at 2026-06-19 04:37:59 on dist-test-slave-n3tq"
server_key: "6df3ffa3ba895b0cda3c03b6f6d6a9a7"
server_key_iv: "bacbf697c06340a1a680fc742e837c93"
server_key_version: "encryptionkey@0"
I20260619 04:37:59.427073 18064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 04:37:59.430709 18078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:59.432077 18064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:37:59.432268 18064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "8647a69f71cb41daa622e72060904cbf"
format_stamp: "Formatted at 2026-06-19 04:37:59 on dist-test-slave-n3tq"
server_key: "6df3ffa3ba895b0cda3c03b6f6d6a9a7"
server_key_iv: "bacbf697c06340a1a680fc742e837c93"
server_key_version: "encryptionkey@0"
I20260619 04:37:59.432500 18064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:37:59.459841 18064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:37:59.460745 18064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:37:59.461007 18064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:37:59.478441 18064 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45057
I20260619 04:37:59.478466 18129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45057 every 8 connection(s)
I20260619 04:37:59.479758 18064 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:37:59.483595 18130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:37:59.487203 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 18064
I20260619 04:37:59.487376 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:37:59.487852 388 external_mini_cluster.cc:1428] Setting key 47d9d58990a37126f016299cdcfc838d
I20260619 04:37:59.496160 18130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf: Bootstrap starting.
I20260619 04:37:59.499420 18130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf: Neither blocks nor log segments found. Creating new log.
I20260619 04:37:59.500562 18130 log.cc:824] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf: Log is configured to *not* fsync() on all Append() calls
I20260619 04:37:59.501295 18133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:37:59.489762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35472 (local address 127.0.97.62:45057)
0619 04:37:59.490783 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0619 04:37:59.490795 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:37:59.490837 (+ 42us) server_negotiation.cc:407] Connection header received
0619 04:37:59.492431 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:37:59.492479 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:37:59.493019 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:37:59.493404 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:37:59.493998 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:59.495366 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:59.496872 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:37:59.497546 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:37:59.497719 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:37:59.498945 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:37:59.498953 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:37:59.498967 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:37:59.499520 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:37:59.499626 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:37:59.499653 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:37:59.499773 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 04:37:59.499982 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 04:37:59.500287 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":347,"threads_started":1}
I20260619 04:37:59.503314 18130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf: No bootstrap required, opened a new log
I20260619 04:37:59.510983 18130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8647a69f71cb41daa622e72060904cbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } }
I20260619 04:37:59.511354 18130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:37:59.511440 18130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8647a69f71cb41daa622e72060904cbf, State: Initialized, Role: FOLLOWER
I20260619 04:37:59.511958 18130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8647a69f71cb41daa622e72060904cbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } }
I20260619 04:37:59.512187 18130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:37:59.512305 18130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:37:59.512444 18130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:37:59.514740 18130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8647a69f71cb41daa622e72060904cbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } }
I20260619 04:37:59.515246 18130 leader_election.cc:302] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8647a69f71cb41daa622e72060904cbf; no voters:
I20260619 04:37:59.515799 18130 leader_election.cc:288] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:37:59.515939 18135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:37:59.516652 18135 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf [term 1 LEADER]: Becoming Leader. State: Replica: 8647a69f71cb41daa622e72060904cbf, State: Running, Role: LEADER
I20260619 04:37:59.517135 18135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8647a69f71cb41daa622e72060904cbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } }
I20260619 04:37:59.517691 18130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:37:59.519788 18137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8647a69f71cb41daa622e72060904cbf. Latest consensus state: current_term: 1 leader_uuid: "8647a69f71cb41daa622e72060904cbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8647a69f71cb41daa622e72060904cbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } }
I20260619 04:37:59.519814 18136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8647a69f71cb41daa622e72060904cbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8647a69f71cb41daa622e72060904cbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } }
I20260619 04:37:59.520253 18136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf [sys.catalog]: This master's current role is: LEADER
I20260619 04:37:59.520327 18137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf [sys.catalog]: This master's current role is: LEADER
I20260619 04:37:59.521492 18140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:37:59.526758 18140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:37:59.535339 18140 catalog_manager.cc:1347] Generated new cluster ID: 4c35d75ad9cb444ba5131f4e2209eee6
I20260619 04:37:59.535465 18140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:37:59.544094 18140 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:37:59.545415 18140 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:37:59.554523 18140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf: Generated new TSK 0
I20260619 04:37:59.555344 18140 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:37:59.566741 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45057
--builtin_ntp_servers=127.0.97.20:36883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 04:37:59.889961 18154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:37:59.890226 18154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:37:59.890348 18154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:37:59.897660 18154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:37:59.897766 18154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:37:59.897897 18154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:37:59.907045 18154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:37:59.908355 18154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:37:59.910148 18154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:37:59.922525 18159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:59.922504 18162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:37:59.922461 18160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:37:59.923151 18154 server_base.cc:1034] running on GCE node
I20260619 04:37:59.923746 18154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:37:59.924707 18154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:37:59.925915 18154 hybrid_clock.cc:648] HybridClock initialized: now 1781843879925870 us; error 39 us; skew 500 ppm
I20260619 04:37:59.926229 18154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:37:59.932945 18154 webserver.cc:460] Webserver started at http://127.0.97.1:40993/ using document root <none> and password file <none>
I20260619 04:37:59.933663 18154 fs_manager.cc:359] Metadata directory not provided
I20260619 04:37:59.933761 18154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:37:59.934080 18154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:37:59.937086 18154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "d104e38d09354142825a83a8cfb59b75"
format_stamp: "Formatted at 2026-06-19 04:37:59 on dist-test-slave-n3tq"
server_key: "a9baf6e00dd30ae5a3383d6e40091a01"
server_key_iv: "5f7cb1464d340be1300586c037e13201"
server_key_version: "encryptionkey@0"
I20260619 04:37:59.937844 18154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d104e38d09354142825a83a8cfb59b75"
format_stamp: "Formatted at 2026-06-19 04:37:59 on dist-test-slave-n3tq"
server_key: "a9baf6e00dd30ae5a3383d6e40091a01"
server_key_iv: "5f7cb1464d340be1300586c037e13201"
server_key_version: "encryptionkey@0"
I20260619 04:37:59.942869 18154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 04:37:59.946528 18168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:59.947860 18154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:37:59.948007 18154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "d104e38d09354142825a83a8cfb59b75"
format_stamp: "Formatted at 2026-06-19 04:37:59 on dist-test-slave-n3tq"
server_key: "a9baf6e00dd30ae5a3383d6e40091a01"
server_key_iv: "5f7cb1464d340be1300586c037e13201"
server_key_version: "encryptionkey@0"
I20260619 04:37:59.948216 18154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:37:59.967959 18154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:37:59.968890 18154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:37:59.969141 18154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:37:59.970111 18154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:37:59.971635 18154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:37:59.971728 18154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:37:59.971833 18154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:37:59.971901 18154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:00.001024 18154 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36291
I20260619 04:38:00.001068 18280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36291 every 8 connection(s)
I20260619 04:38:00.002377 18154 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:38:00.012113 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 18154
I20260619 04:38:00.012293 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:38:00.012853 388 external_mini_cluster.cc:1428] Setting key 8390dcca27f920cf891217446a23302b
I20260619 04:38:00.015504 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:00.004869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52037 (local address 127.0.97.62:45057)
0619 04:38:00.005368 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:00.005375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:00.006765 (+ 1390us) server_negotiation.cc:407] Connection header received
0619 04:38:00.008617 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:00.008621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:00.008706 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:00.008808 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:00.009941 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.010704 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.012745 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.013196 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.013305 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:00.014707 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:00.014711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:00.014714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:00.014738 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:00.014782 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:00.014793 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:00.014867 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:00.015089 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 04:38:00.015188 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":287,"threads_started":1}
I20260619 04:38:00.015661 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45057
--builtin_ntp_servers=127.0.97.20:36883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 04:38:00.016698 18284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:00.005517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.1:52037)
0619 04:38:00.006578 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:00.006622 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:00.008304 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:00.009005 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:00.009014 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:00.009141 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:00.009760 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:00.009778 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.010873 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.010878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:00.012593 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:00.012610 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.013427 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.013433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:00.013649 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:00.014340 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:38:00.014344 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:00.014408 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:38:00.014528 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:00.014884 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:00.014896 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:00.014904 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 04:38:00.015064 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 04:38:00.015506 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":318,"threads_started":1}
I20260619 04:38:00.018277 18281 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45057
I20260619 04:38:00.018606 18281 heartbeater.cc:464] Registering TS with master...
I20260619 04:38:00.019387 18281 heartbeater.cc:511] Master 127.0.97.62:45057 requested a full tablet report, sending...
I20260619 04:38:00.021551 18095 ts_manager.cc:194] Registered new tserver with Master: d104e38d09354142825a83a8cfb59b75 (127.0.97.1:36291)
I20260619 04:38:00.023501 18095 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:52037
W20260619 04:38:00.338281 18285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:00.338574 18285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:00.338701 18285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:00.345999 18285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:00.346118 18285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:00.346261 18285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:38:00.355548 18285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:00.356902 18285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:00.358675 18285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:00.371630 18290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:00.371599 18291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:00.371557 18293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:00.372318 18285 server_base.cc:1034] running on GCE node
I20260619 04:38:00.372946 18285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:00.373895 18285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:00.375108 18285 hybrid_clock.cc:648] HybridClock initialized: now 1781843880375050 us; error 44 us; skew 500 ppm
I20260619 04:38:00.375442 18285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:38:00.377707 18285 webserver.cc:460] Webserver started at http://127.0.97.2:46635/ using document root <none> and password file <none>
I20260619 04:38:00.378424 18285 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:00.378528 18285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:00.378837 18285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:00.381719 18285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "30ab8203c10c4c1e8dbd69be5842353b"
format_stamp: "Formatted at 2026-06-19 04:38:00 on dist-test-slave-n3tq"
server_key: "ade12d612365d7710d0cd9c2189c29cd"
server_key_iv: "919ef17ebd18491245c0011fa833fa7f"
server_key_version: "encryptionkey@0"
I20260619 04:38:00.382493 18285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "30ab8203c10c4c1e8dbd69be5842353b"
format_stamp: "Formatted at 2026-06-19 04:38:00 on dist-test-slave-n3tq"
server_key: "ade12d612365d7710d0cd9c2189c29cd"
server_key_iv: "919ef17ebd18491245c0011fa833fa7f"
server_key_version: "encryptionkey@0"
I20260619 04:38:00.387485 18285 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 04:38:00.390885 18299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:00.392239 18285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:38:00.392408 18285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "30ab8203c10c4c1e8dbd69be5842353b"
format_stamp: "Formatted at 2026-06-19 04:38:00 on dist-test-slave-n3tq"
server_key: "ade12d612365d7710d0cd9c2189c29cd"
server_key_iv: "919ef17ebd18491245c0011fa833fa7f"
server_key_version: "encryptionkey@0"
I20260619 04:38:00.392670 18285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:00.409653 18285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:00.410506 18285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:00.410765 18285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:00.411751 18285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:38:00.413182 18285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:38:00.413292 18285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:00.413396 18285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:38:00.413466 18285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:00.449894 18285 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38909
I20260619 04:38:00.449940 18411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38909 every 8 connection(s)
I20260619 04:38:00.451133 18285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:38:00.460074 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 18285
I20260619 04:38:00.460220 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:38:00.460712 388 external_mini_cluster.cc:1428] Setting key 87cb074b094ffd5b2726f3e832b603e7
I20260619 04:38:00.463569 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:00.453677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51917 (local address 127.0.97.62:45057)
0619 04:38:00.453882 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:00.453888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:00.455284 (+ 1396us) server_negotiation.cc:407] Connection header received
0619 04:38:00.457060 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:00.457064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:00.457135 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:00.457223 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:00.458303 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.459008 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.460710 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.461332 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.461432 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:00.462856 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:00.462861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:00.462864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:00.462881 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:00.462914 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:00.462926 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:00.462991 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:00.463239 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 04:38:00.463316 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 04:38:00.463686 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45057
--builtin_ntp_servers=127.0.97.20:36883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 04:38:00.465070 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:00.454239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.2:51917)
0619 04:38:00.455141 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:00.455177 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:00.456784 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:00.457462 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:00.457470 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:00.457565 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:00.458159 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:00.458177 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.459178 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.459182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:00.460562 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:00.460574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.461560 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.461569 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:00.461833 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:00.462512 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:38:00.462515 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:00.462559 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:38:00.462660 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:00.463017 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:00.463028 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:00.463036 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 04:38:00.463242 (+ 206us) client_negotiation.cc:240] Negotiation successful
0619 04:38:00.463663 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":274,"threads_started":1}
I20260619 04:38:00.466485 18412 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45057
I20260619 04:38:00.466791 18412 heartbeater.cc:464] Registering TS with master...
I20260619 04:38:00.467654 18412 heartbeater.cc:511] Master 127.0.97.62:45057 requested a full tablet report, sending...
I20260619 04:38:00.469215 18095 ts_manager.cc:194] Registered new tserver with Master: 30ab8203c10c4c1e8dbd69be5842353b (127.0.97.2:38909)
I20260619 04:38:00.470196 18095 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:51917
W20260619 04:38:00.786221 18415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:00.786499 18415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:00.786631 18415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:00.793833 18415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:00.793944 18415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:00.794106 18415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:38:00.803164 18415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:00.804479 18415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:00.806178 18415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:00.818369 18420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:00.818337 18423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:00.818399 18421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:00.819257 18415 server_base.cc:1034] running on GCE node
I20260619 04:38:00.819840 18415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:00.820802 18415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:00.822016 18415 hybrid_clock.cc:648] HybridClock initialized: now 1781843880821958 us; error 50 us; skew 500 ppm
I20260619 04:38:00.822317 18415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:38:00.824492 18415 webserver.cc:460] Webserver started at http://127.0.97.3:33715/ using document root <none> and password file <none>
I20260619 04:38:00.825145 18415 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:00.825224 18415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:00.825474 18415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:00.828179 18415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "058e74a69afa4cb597323f51a9bce4de"
format_stamp: "Formatted at 2026-06-19 04:38:00 on dist-test-slave-n3tq"
server_key: "f69cd7ce7911b4533eca91fc9749d9db"
server_key_iv: "62b7ecf6e19a34e4b291ef6896128f6a"
server_key_version: "encryptionkey@0"
I20260619 04:38:00.828931 18415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "058e74a69afa4cb597323f51a9bce4de"
format_stamp: "Formatted at 2026-06-19 04:38:00 on dist-test-slave-n3tq"
server_key: "f69cd7ce7911b4533eca91fc9749d9db"
server_key_iv: "62b7ecf6e19a34e4b291ef6896128f6a"
server_key_version: "encryptionkey@0"
I20260619 04:38:00.833725 18415 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 04:38:00.837121 18429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:00.838410 18415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:38:00.838567 18415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "058e74a69afa4cb597323f51a9bce4de"
format_stamp: "Formatted at 2026-06-19 04:38:00 on dist-test-slave-n3tq"
server_key: "f69cd7ce7911b4533eca91fc9749d9db"
server_key_iv: "62b7ecf6e19a34e4b291ef6896128f6a"
server_key_version: "encryptionkey@0"
I20260619 04:38:00.838765 18415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:00.853336 18415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:00.854120 18415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:00.854363 18415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:00.855261 18415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:38:00.856719 18415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:38:00.856812 18415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:00.856918 18415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:38:00.856984 18415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:00.886447 18415 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44935
I20260619 04:38:00.886487 18541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44935 every 8 connection(s)
I20260619 04:38:00.887715 18415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:38:00.897897 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 18415
I20260619 04:38:00.898097 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:38:00.898514 388 external_mini_cluster.cc:1428] Setting key dcb6fde4533b9e7914e0bbd6bd63f3f1
I20260619 04:38:00.899701 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:00.890053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42117 (local address 127.0.97.62:45057)
0619 04:38:00.890225 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:00.890230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:00.891652 (+ 1422us) server_negotiation.cc:407] Connection header received
0619 04:38:00.893425 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:00.893431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:00.893512 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:00.893597 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:00.894646 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.895360 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.896976 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.897447 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.897560 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:00.898829 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:00.898833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:00.898837 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:00.898857 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:00.898897 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:00.898912 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:00.898996 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:00.899295 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 04:38:00.899389 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 04:38:00.900804 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:00.890627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.3:42117)
0619 04:38:00.891482 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:00.891522 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:00.893155 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:00.893838 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:00.893847 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:00.893933 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:00.894473 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:00.894490 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.895500 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.895504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:00.896842 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:00.896855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.897653 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.897660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:00.897880 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:00.898533 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:38:00.898535 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:00.898572 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:38:00.898660 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:00.899032 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:00.899040 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:00.899072 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 04:38:00.899273 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 04:38:00.899609 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":267,"threads_started":1}
I20260619 04:38:00.901966 18542 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45057
I20260619 04:38:00.902246 18542 heartbeater.cc:464] Registering TS with master...
I20260619 04:38:00.902951 18542 heartbeater.cc:511] Master 127.0.97.62:45057 requested a full tablet report, sending...
I20260619 04:38:00.904238 18095 ts_manager.cc:194] Registered new tserver with Master: 058e74a69afa4cb597323f51a9bce4de (127.0.97.3:44935)
I20260619 04:38:00.905042 18095 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:42117
I20260619 04:38:00.913475 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:38:00.922597 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:00.917319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35480 (local address 127.0.97.62:45057)
0619 04:38:00.917546 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:00.917552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:00.917866 (+ 314us) server_negotiation.cc:407] Connection header received
0619 04:38:00.918002 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:00.918006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:00.918076 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:00.918152 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:00.918555 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.919269 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.920617 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.920994 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.921084 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:00.921985 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:00.921989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:00.921992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:00.922009 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:00.922039 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:00.922050 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:00.922115 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:00.922247 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 04:38:00.922320 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 04:38:00.926095 18095 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:38:00.932714 18095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:38:00.955817 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:00.944538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44935 (local address 127.0.0.1:40598)
0619 04:38:00.947376 (+ 2838us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:00.947393 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:00.947581 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:00.948622 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:00.948631 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:00.948664 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:00.949077 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:00.949090 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.950901 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.950905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:00.953184 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:00.953196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.954871 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.954877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:00.955088 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:00.955140 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 04:38:00.955254 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 04:38:00.955451 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2678,"thread_start_us":2551,"threads_started":1}
I20260619 04:38:00.956144 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:00.944422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38909 (local address 127.0.0.1:37922)
0619 04:38:00.945248 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:00.945298 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:00.945485 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:00.949843 (+ 4358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:00.949846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:00.949870 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:00.950188 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:00.950200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.952021 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.952026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:00.954009 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:00.954021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.955456 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.955461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:00.955624 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:00.955659 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 04:38:00.955722 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 04:38:00.955849 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":831,"threads_started":1}
I20260619 04:38:00.956924 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:00.944603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37922 (local address 127.0.97.2:38909)
0619 04:38:00.949403 (+ 4800us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:00.949410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:00.949435 (+ 25us) server_negotiation.cc:407] Connection header received
0619 04:38:00.949505 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:00.949510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:00.949676 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:00.949833 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:00.950366 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.951864 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.954181 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.955256 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.956163 (+ 907us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:00.956438 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:00.956531 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 04:38:00.956665 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4657,"thread_start_us":4546,"threads_started":1}
I20260619 04:38:00.956998 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:00.945109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40598 (local address 127.0.97.3:44935)
0619 04:38:00.946368 (+ 1259us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:00.946378 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:00.947479 (+ 1101us) server_negotiation.cc:407] Connection header received
0619 04:38:00.947779 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:00.947788 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:00.948052 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:00.948223 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:00.949316 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.950775 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.953365 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.954674 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.954804 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:00.955074 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:00.956428 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0619 04:38:00.956637 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1068,"thread_start_us":927,"threads_started":1}
I20260619 04:38:00.959695 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:00.947457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36291 (local address 127.0.0.1:60126)
0619 04:38:00.948819 (+ 1362us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:00.948835 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:00.948942 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:00.951097 (+ 2155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:00.951102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:00.951131 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:00.951549 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:00.951561 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.953720 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.953734 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:00.955770 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:00.955783 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.958953 (+ 3170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.958960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:00.959125 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:00.959155 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 04:38:00.959248 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 04:38:00.959371 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"mutex_wait_us":3,"thread_start_us":1105,"threads_started":1}
I20260619 04:38:00.959780 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:00.947937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60126 (local address 127.0.97.1:36291)
0619 04:38:00.950503 (+ 2566us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:00.950512 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:00.950537 (+ 25us) server_negotiation.cc:407] Connection header received
0619 04:38:00.950627 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:00.950634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:00.950863 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:00.951786 (+ 923us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:00.952009 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.953575 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.956579 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:00.958758 (+ 2179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:00.958892 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:00.959124 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:00.959298 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 04:38:00.959455 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2410,"thread_start_us":2304,"threads_started":1}
I20260619 04:38:00.960521 18347 tablet_service.cc:1467] Processing CreateTablet for tablet 0d0185a49d784c1a8e8b6ca48a2700eb (DEFAULT_TABLE table=test-table [id=029f99062664488cb94f2e7f523d17e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:38:00.960502 18477 tablet_service.cc:1467] Processing CreateTablet for tablet 0d0185a49d784c1a8e8b6ca48a2700eb (DEFAULT_TABLE table=test-table [id=029f99062664488cb94f2e7f523d17e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:38:00.961865 18347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d0185a49d784c1a8e8b6ca48a2700eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:00.961938 18477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d0185a49d784c1a8e8b6ca48a2700eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:00.963498 18216 tablet_service.cc:1467] Processing CreateTablet for tablet 0d0185a49d784c1a8e8b6ca48a2700eb (DEFAULT_TABLE table=test-table [id=029f99062664488cb94f2e7f523d17e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:38:00.964828 18216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d0185a49d784c1a8e8b6ca48a2700eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:00.972998 18560 tablet_bootstrap.cc:492] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b: Bootstrap starting.
I20260619 04:38:00.976217 18561 tablet_bootstrap.cc:492] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de: Bootstrap starting.
I20260619 04:38:00.976272 18560 tablet_bootstrap.cc:654] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:00.977828 18560 log.cc:824] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:00.978689 18562 tablet_bootstrap.cc:492] T 0d0185a49d784c1a8e8b6ca48a2700eb P d104e38d09354142825a83a8cfb59b75: Bootstrap starting.
I20260619 04:38:00.979838 18561 tablet_bootstrap.cc:654] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:00.980935 18561 log.cc:824] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:00.981691 18562 tablet_bootstrap.cc:654] T 0d0185a49d784c1a8e8b6ca48a2700eb P d104e38d09354142825a83a8cfb59b75: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:00.982647 18560 tablet_bootstrap.cc:492] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b: No bootstrap required, opened a new log
I20260619 04:38:00.982755 18562 log.cc:824] T 0d0185a49d784c1a8e8b6ca48a2700eb P d104e38d09354142825a83a8cfb59b75: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:00.982960 18560 ts_tablet_manager.cc:1397] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260619 04:38:00.984506 18561 tablet_bootstrap.cc:492] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de: No bootstrap required, opened a new log
I20260619 04:38:00.984737 18561 ts_tablet_manager.cc:1397] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 04:38:00.985507 18562 tablet_bootstrap.cc:492] T 0d0185a49d784c1a8e8b6ca48a2700eb P d104e38d09354142825a83a8cfb59b75: No bootstrap required, opened a new log
I20260619 04:38:00.985749 18562 ts_tablet_manager.cc:1397] T 0d0185a49d784c1a8e8b6ca48a2700eb P d104e38d09354142825a83a8cfb59b75: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 04:38:00.991297 18561 raft_consensus.cc:348] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ab8203c10c4c1e8dbd69be5842353b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38909 } } peers { permanent_uuid: "058e74a69afa4cb597323f51a9bce4de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44935 } } peers { permanent_uuid: "d104e38d09354142825a83a8cfb59b75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36291 } }
I20260619 04:38:00.991636 18561 raft_consensus.cc:374] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:00.991760 18561 raft_consensus.cc:729] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 058e74a69afa4cb597323f51a9bce4de, State: Initialized, Role: FOLLOWER
I20260619 04:38:00.992319 18561 consensus_queue.cc:260] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ab8203c10c4c1e8dbd69be5842353b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38909 } } peers { permanent_uuid: "058e74a69afa4cb597323f51a9bce4de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44935 } } peers { permanent_uuid: "d104e38d09354142825a83a8cfb59b75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36291 } }
I20260619 04:38:00.992777 18560 raft_consensus.cc:348] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ab8203c10c4c1e8dbd69be5842353b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38909 } } peers { permanent_uuid: "058e74a69afa4cb597323f51a9bce4de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44935 } } peers { permanent_uuid: "d104e38d09354142825a83a8cfb59b75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36291 } }
I20260619 04:38:00.993359 18542 heartbeater.cc:503] Master 127.0.97.62:45057 was elected leader, sending a full tablet report...
I20260619 04:38:00.994033 18561 ts_tablet_manager.cc:1428] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260619 04:38:00.994385 18560 raft_consensus.cc:374] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:00.994482 18560 raft_consensus.cc:729] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ab8203c10c4c1e8dbd69be5842353b, State: Initialized, Role: FOLLOWER
I20260619 04:38:00.995090 18560 consensus_queue.cc:260] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ab8203c10c4c1e8dbd69be5842353b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38909 } } peers { permanent_uuid: "058e74a69afa4cb597323f51a9bce4de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44935 } } peers { permanent_uuid: "d104e38d09354142825a83a8cfb59b75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36291 } }
I20260619 04:38:00.995857 18562 raft_consensus.cc:348] T 0d0185a49d784c1a8e8b6ca48a2700eb P d104e38d09354142825a83a8cfb59b75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ab8203c10c4c1e8dbd69be5842353b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38909 } } peers { permanent_uuid: "058e74a69afa4cb597323f51a9bce4de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44935 } } peers { permanent_uuid: "d104e38d09354142825a83a8cfb59b75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36291 } }
I20260619 04:38:00.996240 18412 heartbeater.cc:503] Master 127.0.97.62:45057 was elected leader, sending a full tablet report...
I20260619 04:38:00.996306 18562 raft_consensus.cc:374] T 0d0185a49d784c1a8e8b6ca48a2700eb P d104e38d09354142825a83a8cfb59b75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:00.996387 18562 raft_consensus.cc:729] T 0d0185a49d784c1a8e8b6ca48a2700eb P d104e38d09354142825a83a8cfb59b75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d104e38d09354142825a83a8cfb59b75, State: Initialized, Role: FOLLOWER
I20260619 04:38:00.996833 18560 ts_tablet_manager.cc:1428] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b: Time spent starting tablet: real 0.014s user 0.005s sys 0.008s
I20260619 04:38:00.997049 18562 consensus_queue.cc:260] T 0d0185a49d784c1a8e8b6ca48a2700eb P d104e38d09354142825a83a8cfb59b75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ab8203c10c4c1e8dbd69be5842353b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38909 } } peers { permanent_uuid: "058e74a69afa4cb597323f51a9bce4de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44935 } } peers { permanent_uuid: "d104e38d09354142825a83a8cfb59b75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36291 } }
I20260619 04:38:00.998845 18281 heartbeater.cc:503] Master 127.0.97.62:45057 was elected leader, sending a full tablet report...
I20260619 04:38:00.999459 18562 ts_tablet_manager.cc:1428] T 0d0185a49d784c1a8e8b6ca48a2700eb P d104e38d09354142825a83a8cfb59b75: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260619 04:38:01.004539 18282 tablet.cc:2369] T 0d0185a49d784c1a8e8b6ca48a2700eb P d104e38d09354142825a83a8cfb59b75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:38:01.076771 18567 raft_consensus.cc:482] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:38:01.077071 18567 raft_consensus.cc:504] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ab8203c10c4c1e8dbd69be5842353b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38909 } } peers { permanent_uuid: "058e74a69afa4cb597323f51a9bce4de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44935 } } peers { permanent_uuid: "d104e38d09354142825a83a8cfb59b75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36291 } }
I20260619 04:38:01.078701 18567 leader_election.cc:288] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 058e74a69afa4cb597323f51a9bce4de (127.0.97.3:44935), d104e38d09354142825a83a8cfb59b75 (127.0.97.1:36291)
I20260619 04:38:01.083454 18568 raft_consensus.cc:482] T 0d0185a49d784c1a8e8b6ca48a2700eb P d104e38d09354142825a83a8cfb59b75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:38:01.083760 18568 raft_consensus.cc:504] T 0d0185a49d784c1a8e8b6ca48a2700eb P d104e38d09354142825a83a8cfb59b75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ab8203c10c4c1e8dbd69be5842353b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38909 } } peers { permanent_uuid: "058e74a69afa4cb597323f51a9bce4de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44935 } } peers { permanent_uuid: "d104e38d09354142825a83a8cfb59b75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36291 } }
I20260619 04:38:01.085273 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:01.079163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54649 (local address 127.0.97.3:44935)
0619 04:38:01.079332 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:01.079337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:01.079823 (+ 486us) server_negotiation.cc:407] Connection header received
0619 04:38:01.080033 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:01.080039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:01.080121 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:01.080224 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:01.080843 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.081525 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.083521 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.084402 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.084569 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:01.084676 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:01.084921 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 04:38:01.085045 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:38:01.085458 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:01.079078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44935 (local address 127.0.97.2:54649)
0619 04:38:01.079718 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:01.079737 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:01.079884 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:01.080336 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:01.080341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:01.080374 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:01.080697 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:01.080706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.081653 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.081658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:01.083326 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:01.083345 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.084606 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.084612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:01.084772 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:01.084797 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 04:38:01.084874 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 04:38:01.085006 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":1,"thread_start_us":333,"threads_started":1}
I20260619 04:38:01.086442 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:01.079999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51597 (local address 127.0.97.1:36291)
0619 04:38:01.080243 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:01.080249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:01.080627 (+ 378us) server_negotiation.cc:407] Connection header received
0619 04:38:01.080809 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:01.080813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:01.080880 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:01.080951 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:01.081451 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.082145 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.084155 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.085316 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.085432 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:01.085532 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:01.085999 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 04:38:01.086135 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 04:38:01.086607 18497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0185a49d784c1a8e8b6ca48a2700eb" candidate_uuid: "30ab8203c10c4c1e8dbd69be5842353b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058e74a69afa4cb597323f51a9bce4de" is_pre_election: true
I20260619 04:38:01.087088 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:01.079861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36291 (local address 127.0.97.2:51597)
0619 04:38:01.080538 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:01.080554 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:01.080682 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:01.080976 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:01.080980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:01.081002 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:01.081319 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:01.081327 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.082271 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.082276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:01.083961 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:01.083976 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.085651 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.085658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:01.085829 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:01.085861 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 04:38:01.086643 (+ 782us) client_negotiation.cc:240] Negotiation successful
0619 04:38:01.086767 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":25,"thread_start_us":330,"threads_started":1}
I20260619 04:38:01.087167 18497 raft_consensus.cc:2454] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ab8203c10c4c1e8dbd69be5842353b in term 0.
I20260619 04:38:01.087821 18568 leader_election.cc:288] T 0d0185a49d784c1a8e8b6ca48a2700eb P d104e38d09354142825a83a8cfb59b75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ab8203c10c4c1e8dbd69be5842353b (127.0.97.2:38909), 058e74a69afa4cb597323f51a9bce4de (127.0.97.3:44935)
I20260619 04:38:01.088084 18236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0185a49d784c1a8e8b6ca48a2700eb" candidate_uuid: "30ab8203c10c4c1e8dbd69be5842353b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d104e38d09354142825a83a8cfb59b75" is_pre_election: true
I20260619 04:38:01.088584 18236 raft_consensus.cc:2454] T 0d0185a49d784c1a8e8b6ca48a2700eb P d104e38d09354142825a83a8cfb59b75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ab8203c10c4c1e8dbd69be5842353b in term 0.
I20260619 04:38:01.089174 18302 leader_election.cc:302] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ab8203c10c4c1e8dbd69be5842353b, d104e38d09354142825a83a8cfb59b75; no voters:
I20260619 04:38:01.089609 18567 raft_consensus.cc:2781] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:38:01.089828 18567 raft_consensus.cc:482] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:38:01.089941 18567 raft_consensus.cc:3037] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:01.092861 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:01.085979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43903 (local address 127.0.97.2:38909)
0619 04:38:01.086118 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:01.086123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:01.086664 (+ 541us) server_negotiation.cc:407] Connection header received
0619 04:38:01.086922 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:01.086926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:01.086992 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:01.087054 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:01.087858 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.088802 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.091148 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.091977 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.092097 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:01.092187 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:01.092514 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 04:38:01.092631 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 04:38:01.092918 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:01.085927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38909 (local address 127.0.97.1:43903)
0619 04:38:01.086571 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:01.086590 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:01.086759 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:01.087244 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:01.087248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:01.087274 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:01.087692 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:01.087704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.088934 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.088940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:01.090993 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:01.091007 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.092183 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.092190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:01.092330 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:01.092353 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 04:38:01.092434 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 04:38:01.092614 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":327,"threads_started":1}
I20260619 04:38:01.092828 18567 raft_consensus.cc:504] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ab8203c10c4c1e8dbd69be5842353b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38909 } } peers { permanent_uuid: "058e74a69afa4cb597323f51a9bce4de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44935 } } peers { permanent_uuid: "d104e38d09354142825a83a8cfb59b75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36291 } }
I20260619 04:38:01.093624 18567 leader_election.cc:288] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b [CANDIDATE]: Term 1 election: Requested vote from peers 058e74a69afa4cb597323f51a9bce4de (127.0.97.3:44935), d104e38d09354142825a83a8cfb59b75 (127.0.97.1:36291)
I20260619 04:38:01.093833 18367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0185a49d784c1a8e8b6ca48a2700eb" candidate_uuid: "d104e38d09354142825a83a8cfb59b75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ab8203c10c4c1e8dbd69be5842353b" is_pre_election: true
I20260619 04:38:01.094139 18236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0185a49d784c1a8e8b6ca48a2700eb" candidate_uuid: "30ab8203c10c4c1e8dbd69be5842353b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d104e38d09354142825a83a8cfb59b75"
I20260619 04:38:01.094323 18236 raft_consensus.cc:3037] T 0d0185a49d784c1a8e8b6ca48a2700eb P d104e38d09354142825a83a8cfb59b75 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:01.094386 18367 raft_consensus.cc:2379] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d104e38d09354142825a83a8cfb59b75 in current term 1: Already voted for candidate 30ab8203c10c4c1e8dbd69be5842353b in this term.
I20260619 04:38:01.094698 18497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0185a49d784c1a8e8b6ca48a2700eb" candidate_uuid: "30ab8203c10c4c1e8dbd69be5842353b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058e74a69afa4cb597323f51a9bce4de"
I20260619 04:38:01.094899 18497 raft_consensus.cc:3037] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:01.095482 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:01.086945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44935 (local address 127.0.97.1:57167)
0619 04:38:01.088485 (+ 1540us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:01.088501 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:01.088601 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:01.089250 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:01.089255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:01.089282 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:01.089680 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:01.089693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.091088 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.091094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:01.093074 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:01.093087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.094748 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.094754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:01.094918 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:01.094951 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 04:38:01.095059 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 04:38:01.095189 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1382,"thread_start_us":1164,"threads_started":1}
I20260619 04:38:01.095498 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:01.088792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57167 (local address 127.0.97.3:44935)
0619 04:38:01.089021 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:01.089026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:01.089041 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:38:01.089089 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:01.089092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:01.089141 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:01.089213 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:01.089898 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.090945 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.093284 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.094418 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.094524 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:01.094623 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:01.095090 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 04:38:01.095231 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260619 04:38:01.097291 18236 raft_consensus.cc:2454] T 0d0185a49d784c1a8e8b6ca48a2700eb P d104e38d09354142825a83a8cfb59b75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ab8203c10c4c1e8dbd69be5842353b in term 1.
I20260619 04:38:01.097564 18497 raft_consensus.cc:2454] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ab8203c10c4c1e8dbd69be5842353b in term 1.
I20260619 04:38:01.097946 18302 leader_election.cc:302] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058e74a69afa4cb597323f51a9bce4de, 30ab8203c10c4c1e8dbd69be5842353b; no voters:
I20260619 04:38:01.098182 18497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0185a49d784c1a8e8b6ca48a2700eb" candidate_uuid: "d104e38d09354142825a83a8cfb59b75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058e74a69afa4cb597323f51a9bce4de" is_pre_election: true
I20260619 04:38:01.098227 18567 raft_consensus.cc:2781] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:01.098424 18497 raft_consensus.cc:2379] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d104e38d09354142825a83a8cfb59b75 in current term 1: Already voted for candidate 30ab8203c10c4c1e8dbd69be5842353b in this term.
I20260619 04:38:01.098637 18567 raft_consensus.cc:686] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b [term 1 LEADER]: Becoming Leader. State: Replica: 30ab8203c10c4c1e8dbd69be5842353b, State: Running, Role: LEADER
I20260619 04:38:01.099026 18171 leader_election.cc:302] T 0d0185a49d784c1a8e8b6ca48a2700eb P d104e38d09354142825a83a8cfb59b75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d104e38d09354142825a83a8cfb59b75; no voters: 058e74a69afa4cb597323f51a9bce4de, 30ab8203c10c4c1e8dbd69be5842353b
I20260619 04:38:01.099136 18567 consensus_queue.cc:237] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ab8203c10c4c1e8dbd69be5842353b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38909 } } peers { permanent_uuid: "058e74a69afa4cb597323f51a9bce4de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44935 } } peers { permanent_uuid: "d104e38d09354142825a83a8cfb59b75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36291 } }
I20260619 04:38:01.099359 18568 raft_consensus.cc:2726] T 0d0185a49d784c1a8e8b6ca48a2700eb P d104e38d09354142825a83a8cfb59b75 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 04:38:01.103442 18095 catalog_manager.cc:5495] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b reported cstate change: term changed from 0 to 1, leader changed from <none> to 30ab8203c10c4c1e8dbd69be5842353b (127.0.97.2). New cstate: current_term: 1 leader_uuid: "30ab8203c10c4c1e8dbd69be5842353b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ab8203c10c4c1e8dbd69be5842353b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "058e74a69afa4cb597323f51a9bce4de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d104e38d09354142825a83a8cfb59b75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36291 } health_report { overall_health: UNKNOWN } } }
W20260619 04:38:01.139284 18543 tablet.cc:2369] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:38:01.145421 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:01.140182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37926 (local address 127.0.97.2:38909)
0619 04:38:01.140353 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:01.140359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:01.140379 (+ 20us) server_negotiation.cc:407] Connection header received
0619 04:38:01.140537 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:01.140541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:01.140636 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:01.140761 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:01.141113 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.141859 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.143321 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.143734 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.143842 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:01.144317 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:01.144833 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:01.144974 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:01.145057 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 04:38:01.145137 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 04:38:01.153231 18497 raft_consensus.cc:1261] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de [term 1 FOLLOWER]: Refusing update from remote peer 30ab8203c10c4c1e8dbd69be5842353b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:38:01.153275 18236 raft_consensus.cc:1261] T 0d0185a49d784c1a8e8b6ca48a2700eb P d104e38d09354142825a83a8cfb59b75 [term 1 FOLLOWER]: Refusing update from remote peer 30ab8203c10c4c1e8dbd69be5842353b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:38:01.154043 18573 consensus_queue.cc:1035] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b [LEADER]: Connected to new peer: Peer: permanent_uuid: "058e74a69afa4cb597323f51a9bce4de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:38:01.154424 18567 consensus_queue.cc:1035] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d104e38d09354142825a83a8cfb59b75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:38:01.164948 18576 mvcc.cc:205] Tried to move back new op lower bound from 7298432537194479616 to 7298432536986251264. Current Snapshot: MvccSnapshot[applied={T|T < 7298432537194479616}]
I20260619 04:38:01.170089 18577 mvcc.cc:205] Tried to move back new op lower bound from 7298432537194479616 to 7298432536986251264. Current Snapshot: MvccSnapshot[applied={T|T < 7298432537194479616}]
I20260619 04:38:01.179488 18094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:38:01.179775 18094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:38:01.183485 18094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8647a69f71cb41daa622e72060904cbf: Sending DeleteTablet for 3 replicas of tablet 0d0185a49d784c1a8e8b6ca48a2700eb
I20260619 04:38:01.184808 18347 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d0185a49d784c1a8e8b6ca48a2700eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:38:01 UTC) from {username='slave'} at 127.0.0.1:37922
I20260619 04:38:01.184950 18216 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d0185a49d784c1a8e8b6ca48a2700eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:38:01 UTC) from {username='slave'} at 127.0.0.1:60126
I20260619 04:38:01.185590 18477 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d0185a49d784c1a8e8b6ca48a2700eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:38:01 UTC) from {username='slave'} at 127.0.0.1:40598
I20260619 04:38:01.185819 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 18154
I20260619 04:38:01.206284 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:01.187148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60142 (local address 127.0.97.1:36291)
0619 04:38:01.187318 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:01.187324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:01.197440 (+ 10116us) server_negotiation.cc:407] Connection header received
0619 04:38:01.198023 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:01.198030 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:01.198181 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:01.198267 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:01.198933 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.200154 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.202205 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.202851 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.202947 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:01.204446 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:01.204498 (+ 52us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:01.204516 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:01.205323 (+ 807us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:01.205389 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:01.205415 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:01.205509 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:01.205749 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 04:38:01.205874 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:38:01.207973 18276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:01.208400 18588 tablet_replica.cc:330] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de: stopping tablet replica
I20260619 04:38:01.208933 18588 raft_consensus.cc:2229] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:38:01.209787 18586 tablet_replica.cc:330] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b: stopping tablet replica
I20260619 04:38:01.210466 18588 raft_consensus.cc:2258] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:38:01.211130 18586 raft_consensus.cc:2229] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:38:01.212291 18586 raft_consensus.cc:2258] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:38:01.214336 18588 ts_tablet_manager.cc:1905] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:38:01.216359 18586 ts_tablet_manager.cc:1905] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:38:01.229671 18588 ts_tablet_manager.cc:1918] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:38:01.229920 18588 log.cc:1196] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/0d0185a49d784c1a8e8b6ca48a2700eb
I20260619 04:38:01.230445 18588 ts_tablet_manager.cc:1939] T 0d0185a49d784c1a8e8b6ca48a2700eb P 058e74a69afa4cb597323f51a9bce4de: Deleting consensus metadata
I20260619 04:38:01.232506 18081 catalog_manager.cc:4841] TS 058e74a69afa4cb597323f51a9bce4de (127.0.97.3:44935): tablet 0d0185a49d784c1a8e8b6ca48a2700eb (table test-table [id=029f99062664488cb94f2e7f523d17e8]) successfully deleted
I20260619 04:38:01.233265 18586 ts_tablet_manager.cc:1918] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:38:01.233412 18586 log.cc:1196] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/0d0185a49d784c1a8e8b6ca48a2700eb
I20260619 04:38:01.233843 18586 ts_tablet_manager.cc:1939] T 0d0185a49d784c1a8e8b6ca48a2700eb P 30ab8203c10c4c1e8dbd69be5842353b: Deleting consensus metadata
I20260619 04:38:01.235296 18080 catalog_manager.cc:4841] TS 30ab8203c10c4c1e8dbd69be5842353b (127.0.97.2:38909): tablet 0d0185a49d784c1a8e8b6ca48a2700eb (table test-table [id=029f99062664488cb94f2e7f523d17e8]) successfully deleted
I20260619 04:38:01.356231 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 18154
I20260619 04:38:01.356474 18587 tablet_replica.cc:330] T 0d0185a49d784c1a8e8b6ca48a2700eb P d104e38d09354142825a83a8cfb59b75: stopping tablet replica
W20260619 04:38:01.370769 18081 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:36291 (error 108)
I20260619 04:38:01.371158 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 18285
I20260619 04:38:01.372912 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:01.372351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36291 (local address 127.0.0.1:60148)
0619 04:38:01.372581 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:01.372727 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 04:38:01.373198 18081 catalog_manager.cc:4584] TS d104e38d09354142825a83a8cfb59b75 (127.0.97.1:36291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0d0185a49d784c1a8e8b6ca48a2700eb: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36291: connect: Connection refused (error 111)
I20260619 04:38:01.378479 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:01.372063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37936 (local address 127.0.97.2:38909)
0619 04:38:01.372234 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:01.372241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:01.372261 (+ 20us) server_negotiation.cc:407] Connection header received
0619 04:38:01.372382 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:01.372388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:01.372646 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:01.372754 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:01.373304 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.374085 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.375431 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.375875 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.375997 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:01.377100 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:01.377109 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:01.377123 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:01.377745 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:01.377794 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:01.377814 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:01.377913 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:01.378066 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 04:38:01.378170 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 04:38:01.379361 18407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:01.426764 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:01.426216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36291 (local address 127.0.0.1:60152)
0619 04:38:01.426447 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:01.426560 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 04:38:01.427796 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:01.427426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36291 (local address 127.0.0.1:60154)
0619 04:38:01.427562 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:01.427645 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 04:38:01.488615 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:01.487982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36291 (local address 127.0.0.1:60158)
0619 04:38:01.488263 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:01.488381 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260619 04:38:01.489817 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:01.489406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36291 (local address 127.0.0.1:60168)
0619 04:38:01.489574 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:01.489670 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 04:38:01.492939 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 18285
I20260619 04:38:01.507721 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 18415
I20260619 04:38:01.514173 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:01.508517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40602 (local address 127.0.97.3:44935)
0619 04:38:01.508701 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:01.508706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:01.508725 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:38:01.508856 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:01.508860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:01.508941 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:01.509030 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:01.509405 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.510157 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.511392 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:01.511782 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:01.511874 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:01.512937 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:01.512945 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:01.512961 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:01.513529 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:01.513576 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:01.513594 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:01.513682 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:01.513785 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 04:38:01.513862 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:38:01.514791 18537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:01.570632 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:01.570076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36291 (local address 127.0.0.1:60180)
0619 04:38:01.570338 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:01.570454 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260619 04:38:01.571650 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:01.571290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36291 (local address 127.0.0.1:60196)
0619 04:38:01.571464 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:01.571542 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 04:38:01.602224 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 18415
I20260619 04:38:01.615561 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 18064
I20260619 04:38:01.616679 18125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:01.726027 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 18064
2026-06-19T04:38:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2744 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:38:01 dist-test-slave-n3tq krb5kdc[18597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:38:01 dist-test-slave-n3tq krb5kdc[18597](info): set up 2 sockets
Jun 19 04:38:01 dist-test-slave-n3tq krb5kdc[18597](info): commencing operation
krb5kdc: starting...
W20260619 04:38:02.740233 388 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:38:02 dist-test-slave-n3tq krb5kdc[18597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:38:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:38:02.969234 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43829
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:03.282476 18610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:03.282750 18610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:03.282874 18610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:03.290015 18610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:03.290123 18610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:03.290179 18610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:03.290256 18610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:03.290303 18610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:03.299898 18610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:03.301195 18610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:03.302471 18610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:03.313413 18615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:03.313386 18618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:03.313355 18616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:03.314167 18610 server_base.cc:1034] running on GCE node
I20260619 04:38:03.314842 18610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:03.316118 18610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:03.317305 18610 hybrid_clock.cc:648] HybridClock initialized: now 1781843883317254 us; error 40 us; skew 500 ppm
I20260619 04:38:03.317621 18610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:03 dist-test-slave-n3tq krb5kdc[18597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:03.320489 18610 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:38:03.321837 18610 webserver.cc:460] Webserver started at http://127.0.97.62:38489/ using document root <none> and password file <none>
I20260619 04:38:03.322463 18610 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:03.322556 18610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:03.322794 18610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:03.325657 18610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "6f85d698695f4e87b6da8f6b1092e682"
format_stamp: "Formatted at 2026-06-19 04:38:03 on dist-test-slave-n3tq"
server_key: "0f6059df2e31897cfcd377d14cd32b24"
server_key_iv: "9bd0fc2abce96e01ef72fadd8f8dc7a6"
server_key_version: "encryptionkey@0"
I20260619 04:38:03.326346 18610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "6f85d698695f4e87b6da8f6b1092e682"
format_stamp: "Formatted at 2026-06-19 04:38:03 on dist-test-slave-n3tq"
server_key: "0f6059df2e31897cfcd377d14cd32b24"
server_key_iv: "9bd0fc2abce96e01ef72fadd8f8dc7a6"
server_key_version: "encryptionkey@0"
I20260619 04:38:03.331208 18610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:38:03.334410 18625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:03.335603 18610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 04:38:03.335732 18610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "6f85d698695f4e87b6da8f6b1092e682"
format_stamp: "Formatted at 2026-06-19 04:38:03 on dist-test-slave-n3tq"
server_key: "0f6059df2e31897cfcd377d14cd32b24"
server_key_iv: "9bd0fc2abce96e01ef72fadd8f8dc7a6"
server_key_version: "encryptionkey@0"
I20260619 04:38:03.335911 18610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:03.364087 18610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:03.368305 18610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:03.368610 18610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:03.385944 18610 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43829
I20260619 04:38:03.385963 18676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43829 every 8 connection(s)
I20260619 04:38:03.387312 18610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:03.390538 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 18610
I20260619 04:38:03.390714 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:03.391204 18677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:03.391247 388 external_mini_cluster.cc:1428] Setting key 254a73f5041ba356d6f95dfb66f9010e
I20260619 04:38:03.405200 18677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682: Bootstrap starting.
Jun 19 04:38:03 dist-test-slave-n3tq krb5kdc[18597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:38:03.408275 18677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:03.409322 18677 log.cc:824] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:03.411974 18677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682: No bootstrap required, opened a new log
I20260619 04:38:03.413553 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:03.393083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38876 (local address 127.0.97.62:43829)
0619 04:38:03.393843 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:03.393856 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:03.393897 (+ 41us) server_negotiation.cc:407] Connection header received
0619 04:38:03.395781 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:03.395804 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:03.396200 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:03.396573 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:03.398261 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:03.399578 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:03.401375 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:03.402105 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:03.402244 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:03.406872 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:03.406880 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:03.406894 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:03.406929 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:03.409757 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:03.410410 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:03.410417 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:03.410423 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:03.410585 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:03.410925 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:03.410929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:03.410931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:03.411421 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:03.411702 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:03.412131 (+ 429us) server_negotiation.cc:299] Negotiation successful
0619 04:38:03.412576 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":340,"threads_started":1}
I20260619 04:38:03.419516 18677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f85d698695f4e87b6da8f6b1092e682" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43829 } }
I20260619 04:38:03.419871 18677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:03.419970 18677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f85d698695f4e87b6da8f6b1092e682, State: Initialized, Role: FOLLOWER
I20260619 04:38:03.420490 18677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f85d698695f4e87b6da8f6b1092e682" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43829 } }
I20260619 04:38:03.420756 18677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:38:03.420898 18677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:38:03.421063 18677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:03.423285 18677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f85d698695f4e87b6da8f6b1092e682" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43829 } }
I20260619 04:38:03.423768 18677 leader_election.cc:302] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f85d698695f4e87b6da8f6b1092e682; no voters:
I20260619 04:38:03.424430 18677 leader_election.cc:288] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:38:03.424571 18682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:03.425194 18682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682 [term 1 LEADER]: Becoming Leader. State: Replica: 6f85d698695f4e87b6da8f6b1092e682, State: Running, Role: LEADER
I20260619 04:38:03.425730 18682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f85d698695f4e87b6da8f6b1092e682" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43829 } }
I20260619 04:38:03.426426 18677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:03.428442 18683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f85d698695f4e87b6da8f6b1092e682" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f85d698695f4e87b6da8f6b1092e682" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43829 } } }
I20260619 04:38:03.428938 18683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:03.428413 18684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f85d698695f4e87b6da8f6b1092e682. Latest consensus state: current_term: 1 leader_uuid: "6f85d698695f4e87b6da8f6b1092e682" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f85d698695f4e87b6da8f6b1092e682" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43829 } } }
I20260619 04:38:03.430056 18684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:03.430580 18691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:03.435555 18691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:03.444491 18691 catalog_manager.cc:1347] Generated new cluster ID: 71db6c2ec66545e487848c714989cc2a
I20260619 04:38:03.444599 18691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:03.478298 18691 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:03.479826 18691 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:03.494748 18691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682: Generated new TSK 0
I20260619 04:38:03.495575 18691 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 04:38:03.549882 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43829
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:03.869208 18703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:03.869515 18703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:03.869609 18703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:03.877138 18703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:03.877255 18703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:03.877375 18703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:38:03.887207 18703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:03.888523 18703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:03.889799 18703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:03.902236 18709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:03.902279 18711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:03.902299 18708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:03.903403 18703 server_base.cc:1034] running on GCE node
I20260619 04:38:03.904031 18703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:03.905052 18703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:03.906255 18703 hybrid_clock.cc:648] HybridClock initialized: now 1781843883906207 us; error 57 us; skew 500 ppm
I20260619 04:38:03.906548 18703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:03 dist-test-slave-n3tq krb5kdc[18597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:03.909602 18703 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:38:03.911029 18703 webserver.cc:460] Webserver started at http://127.0.97.1:40489/ using document root <none> and password file <none>
I20260619 04:38:03.911679 18703 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:03.911777 18703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:03.912070 18703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:03.915091 18703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "bd1c51fa48e84198a35c39dad2257acc"
format_stamp: "Formatted at 2026-06-19 04:38:03 on dist-test-slave-n3tq"
server_key: "22c3aff58bdf4331f5ab665c53702281"
server_key_iv: "f3753fbe62d386edacfd39f761e79808"
server_key_version: "encryptionkey@0"
I20260619 04:38:03.915838 18703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bd1c51fa48e84198a35c39dad2257acc"
format_stamp: "Formatted at 2026-06-19 04:38:03 on dist-test-slave-n3tq"
server_key: "22c3aff58bdf4331f5ab665c53702281"
server_key_iv: "f3753fbe62d386edacfd39f761e79808"
server_key_version: "encryptionkey@0"
I20260619 04:38:03.920929 18703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 04:38:03.924623 18718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:03.925992 18703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:38:03.926141 18703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "bd1c51fa48e84198a35c39dad2257acc"
format_stamp: "Formatted at 2026-06-19 04:38:03 on dist-test-slave-n3tq"
server_key: "22c3aff58bdf4331f5ab665c53702281"
server_key_iv: "f3753fbe62d386edacfd39f761e79808"
server_key_version: "encryptionkey@0"
I20260619 04:38:03.926352 18703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:03.943554 18703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:03.947815 18703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:03.948061 18703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:03.949013 18703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:38:03.950325 18703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:38:03.950397 18703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:03.950482 18703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:38:03.950525 18703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:03.979570 18703 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32987
I20260619 04:38:03.979614 18830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32987 every 8 connection(s)
I20260619 04:38:03.980849 18703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:38:03.981810 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 18703
I20260619 04:38:03.981997 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:38:03.982537 388 external_mini_cluster.cc:1428] Setting key 08e985dfa1f5691bdf814c76795a08ab
Jun 19 04:38:03 dist-test-slave-n3tq krb5kdc[18597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:38:04.002238 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:03.983495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51129 (local address 127.0.97.62:43829)
0619 04:38:03.983922 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:03.983927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:03.985139 (+ 1212us) server_negotiation.cc:407] Connection header received
0619 04:38:03.986950 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:03.986955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:03.987031 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:03.987134 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:03.988546 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:03.989572 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:03.991083 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:03.991659 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:03.991773 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:03.995911 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:03.995916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:03.995919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:03.995953 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:03.998985 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:03.999766 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:03.999771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:03.999774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:03.999926 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:04.000357 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:04.000363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:04.000366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:04.000751 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:04.000914 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:04.001623 (+ 709us) server_negotiation.cc:299] Negotiation successful
0619 04:38:04.001866 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":221,"threads_started":1}
I20260619 04:38:04.003060 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:03.984107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43829 (local address 127.0.97.1:51129)
0619 04:38:03.984962 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:03.985014 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:03.986682 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:03.987300 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:03.987308 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:03.987819 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:03.988362 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:03.988379 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:03.989721 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:03.989724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:03.990952 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:03.990964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:03.991899 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:03.991907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:03.992129 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:03.992797 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:03.992823 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:03.995655 (+ 2832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:03.999146 (+ 3491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:03.999153 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:03.999167 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:03.999632 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:04.000013 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:04.000016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:04.000019 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:04.000205 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:04.000890 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:04.000897 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:04.001249 (+ 352us) client_negotiation.cc:769] Sending connection context
0619 04:38:04.001502 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 04:38:04.001906 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":341,"threads_started":1}
I20260619 04:38:04.004901 18831 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43829
I20260619 04:38:04.005200 18831 heartbeater.cc:464] Registering TS with master...
I20260619 04:38:04.005926 18831 heartbeater.cc:511] Master 127.0.97.62:43829 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 04:38:04.008080 18642 ts_manager.cc:194] Registered new tserver with Master: bd1c51fa48e84198a35c39dad2257acc (127.0.97.1:32987)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 04:38:04.009795 18642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 04:38:04.034710 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43829
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:04.349431 18837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:04.349695 18837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:04.349812 18837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:04.356961 18837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:04.357069 18837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:04.357206 18837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:38:04.366609 18837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:04.367832 18837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:04.369110 18837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:04.381568 18842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:04.381541 18845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:04.381623 18843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:04.382292 18837 server_base.cc:1034] running on GCE node
I20260619 04:38:04.382860 18837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:04.383749 18837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:04.384963 18837 hybrid_clock.cc:648] HybridClock initialized: now 1781843884384904 us; error 45 us; skew 500 ppm
I20260619 04:38:04.385290 18837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:04 dist-test-slave-n3tq krb5kdc[18597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:04.388592 18837 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:38:04.390254 18837 webserver.cc:460] Webserver started at http://127.0.97.2:40901/ using document root <none> and password file <none>
I20260619 04:38:04.391064 18837 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:04.391191 18837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:04.391512 18837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:04.395186 18837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "8fd86f2f9f6d4423868542da14492333"
format_stamp: "Formatted at 2026-06-19 04:38:04 on dist-test-slave-n3tq"
server_key: "c981ec3a890d74cef49b85038a7abe23"
server_key_iv: "5ec54e992249aff3624512237167c4d3"
server_key_version: "encryptionkey@0"
I20260619 04:38:04.396128 18837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8fd86f2f9f6d4423868542da14492333"
format_stamp: "Formatted at 2026-06-19 04:38:04 on dist-test-slave-n3tq"
server_key: "c981ec3a890d74cef49b85038a7abe23"
server_key_iv: "5ec54e992249aff3624512237167c4d3"
server_key_version: "encryptionkey@0"
I20260619 04:38:04.402405 18837 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 04:38:04.406339 18852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:04.407640 18837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 04:38:04.407783 18837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "8fd86f2f9f6d4423868542da14492333"
format_stamp: "Formatted at 2026-06-19 04:38:04 on dist-test-slave-n3tq"
server_key: "c981ec3a890d74cef49b85038a7abe23"
server_key_iv: "5ec54e992249aff3624512237167c4d3"
server_key_version: "encryptionkey@0"
I20260619 04:38:04.407965 18837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:04.438146 18837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:04.442327 18837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:04.442601 18837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:04.443542 18837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:38:04.444905 18837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:38:04.444975 18837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:04.445084 18837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:38:04.445137 18837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:04.474695 18837 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32993
I20260619 04:38:04.474740 18964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32993 every 8 connection(s)
I20260619 04:38:04.475939 18837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:38:04.476492 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 18837
I20260619 04:38:04.476663 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:38:04.477238 388 external_mini_cluster.cc:1428] Setting key e3abc610a3275ee4deb1af29a0509409
Jun 19 04:38:04 dist-test-slave-n3tq krb5kdc[18597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:38:04.497354 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:04.478512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50261 (local address 127.0.97.62:43829)
0619 04:38:04.478691 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:04.478696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:04.480010 (+ 1314us) server_negotiation.cc:407] Connection header received
0619 04:38:04.482423 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:04.482427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:04.482495 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:04.482572 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:04.484404 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:04.485158 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:04.487272 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:04.487686 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:04.487823 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:04.491402 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:04.491408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:04.491413 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:04.491452 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:04.494412 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:04.495114 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:04.495118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:04.495120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:04.495252 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:04.495659 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:04.495663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:04.495665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:04.495930 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:04.496067 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:04.496779 (+ 712us) server_negotiation.cc:299] Negotiation successful
0619 04:38:04.497020 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:38:04.498399 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:04.479033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43829 (local address 127.0.97.2:50261)
0619 04:38:04.479831 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:04.479888 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:04.482125 (+ 2237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:04.482823 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:04.482833 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:04.483471 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:04.484211 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:04.484233 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:04.485294 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:04.485299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:04.487046 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:04.487061 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:04.487896 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:04.487902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:04.488170 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:04.488888 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:04.488910 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:04.491145 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:04.494547 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:04.494553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:04.494567 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:04.494988 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:04.495367 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:04.495370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:04.495372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:04.495553 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:04.496052 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:04.496060 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:04.496388 (+ 328us) client_negotiation.cc:769] Sending connection context
0619 04:38:04.496673 (+ 285us) client_negotiation.cc:240] Negotiation successful
0619 04:38:04.497238 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":317,"threads_started":1}
I20260619 04:38:04.499848 18965 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43829
I20260619 04:38:04.500191 18965 heartbeater.cc:464] Registering TS with master...
I20260619 04:38:04.501097 18965 heartbeater.cc:511] Master 127.0.97.62:43829 requested a full tablet report, sending...
I20260619 04:38:04.502753 18642 ts_manager.cc:194] Registered new tserver with Master: 8fd86f2f9f6d4423868542da14492333 (127.0.97.2:32993)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 04:38:04.503770 18642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 04:38:04.532287 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43829
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:04.848001 18970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:04.848307 18970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:04.848435 18970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:04.855666 18970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:04.855800 18970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:04.855957 18970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:38:04.865705 18970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:04.866947 18970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:04.868227 18970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:04.880828 18975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:04.880836 18976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:04.880926 18978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:04.881783 18970 server_base.cc:1034] running on GCE node
I20260619 04:38:04.882334 18970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:04.883271 18970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:04.884485 18970 hybrid_clock.cc:648] HybridClock initialized: now 1781843884884403 us; error 54 us; skew 500 ppm
I20260619 04:38:04.884821 18970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:04 dist-test-slave-n3tq krb5kdc[18597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:04.887682 18970 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:38:04.889134 18970 webserver.cc:460] Webserver started at http://127.0.97.3:39879/ using document root <none> and password file <none>
I20260619 04:38:04.889791 18970 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:04.889894 18970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:04.890177 18970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:04.893119 18970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "07d4476fb4484bada3b074fcb67ed3ec"
format_stamp: "Formatted at 2026-06-19 04:38:04 on dist-test-slave-n3tq"
server_key: "43d3ae1a2af3ff8854bcc0b2cfdafd2e"
server_key_iv: "3b07afddc7185fc85dd1cc7552b5b564"
server_key_version: "encryptionkey@0"
I20260619 04:38:04.893934 18970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "07d4476fb4484bada3b074fcb67ed3ec"
format_stamp: "Formatted at 2026-06-19 04:38:04 on dist-test-slave-n3tq"
server_key: "43d3ae1a2af3ff8854bcc0b2cfdafd2e"
server_key_iv: "3b07afddc7185fc85dd1cc7552b5b564"
server_key_version: "encryptionkey@0"
I20260619 04:38:04.899010 18970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:38:04.902320 18985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:04.903709 18970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:38:04.903872 18970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "07d4476fb4484bada3b074fcb67ed3ec"
format_stamp: "Formatted at 2026-06-19 04:38:04 on dist-test-slave-n3tq"
server_key: "43d3ae1a2af3ff8854bcc0b2cfdafd2e"
server_key_iv: "3b07afddc7185fc85dd1cc7552b5b564"
server_key_version: "encryptionkey@0"
I20260619 04:38:04.904062 18970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:04.920749 18970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:04.924508 18970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:04.924958 18970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:04.925923 18970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:38:04.927229 18970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:38:04.927301 18970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:04.927420 18970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:38:04.927481 18970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:04.956800 18970 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46113
I20260619 04:38:04.956849 19097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46113 every 8 connection(s)
I20260619 04:38:04.958063 18970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:38:04.964687 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 18970
I20260619 04:38:04.964846 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:38:04.965242 388 external_mini_cluster.cc:1428] Setting key 69f9843000d9d5a27e96ea98e5f0d704
Jun 19 04:38:04 dist-test-slave-n3tq krb5kdc[18597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:38:04.984978 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:04.960508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41021 (local address 127.0.97.62:43829)
0619 04:38:04.960690 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:04.960694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:04.961855 (+ 1161us) server_negotiation.cc:407] Connection header received
0619 04:38:04.963635 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:04.963640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:04.963722 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:04.963815 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:04.965370 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:04.966148 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:04.967623 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:04.968014 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:04.968117 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:04.979293 (+ 11176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:04.979299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:04.979304 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:04.979344 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:04.981943 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:04.982782 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:04.982785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:04.982788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:04.982910 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:04.983331 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:04.983334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:04.983336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:04.983616 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:04.983728 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:04.984557 (+ 829us) server_negotiation.cc:299] Negotiation successful
0619 04:38:04.984727 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 04:38:04.985816 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:04.961008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43829 (local address 127.0.97.3:41021)
0619 04:38:04.961685 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:04.961748 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:04.963376 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:04.964051 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:04.964059 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:04.964600 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:04.965210 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:04.965230 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:04.966296 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:04.966300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:04.967495 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:04.967507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:04.968214 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:04.968222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:04.968443 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:04.976728 (+ 8285us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:04.976753 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:04.979027 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:04.982085 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:04.982095 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:04.982112 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:04.982670 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:04.982992 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:04.982997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:04.983001 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:04.983228 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:04.983740 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:04.983748 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:04.984131 (+ 383us) client_negotiation.cc:769] Sending connection context
0619 04:38:04.984427 (+ 296us) client_negotiation.cc:240] Negotiation successful
0619 04:38:04.984843 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":276,"threads_started":1}
I20260619 04:38:04.987099 19098 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43829
I20260619 04:38:04.987396 19098 heartbeater.cc:464] Registering TS with master...
I20260619 04:38:04.988051 19098 heartbeater.cc:511] Master 127.0.97.62:43829 requested a full tablet report, sending...
I20260619 04:38:04.989367 18642 ts_manager.cc:194] Registered new tserver with Master: 07d4476fb4484bada3b074fcb67ed3ec (127.0.97.3:46113)
I20260619 04:38:04.990120 18642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41021
I20260619 04:38:04.991798 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:38:05.007893 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:04.995853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38888 (local address 127.0.97.62:43829)
0619 04:38:04.996023 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:04.996028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:04.996372 (+ 344us) server_negotiation.cc:407] Connection header received
0619 04:38:04.996542 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:04.996546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:04.996606 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:04.996689 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:04.997982 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:04.998687 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.000005 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.000430 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.000575 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:05.002964 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:05.002969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:05.002972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:05.003002 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:05.005473 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:05.006149 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:05.006153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:05.006156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:05.006265 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:05.006599 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:05.006602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:05.006604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:05.006921 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:05.007121 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:05.007490 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 04:38:05.007619 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:38:05.012604 18831 heartbeater.cc:503] Master 127.0.97.62:43829 was elected leader, sending a full tablet report...
I20260619 04:38:05.011615 18642 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:38:05.018627 18642 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:38:05.050065 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:05.031245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37622 (local address 127.0.97.1:32987)
0619 04:38:05.041053 (+ 9808us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:05.041062 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:05.041850 (+ 788us) server_negotiation.cc:407] Connection header received
0619 04:38:05.042012 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:05.042021 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:05.042327 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:05.042543 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:05.043508 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.045120 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.047830 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.049011 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.049134 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:05.049372 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:05.049584 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 04:38:05.049744 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9602,"thread_start_us":9815,"threads_started":1}
I20260619 04:38:05.050155 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:05.030583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32987 (local address 127.0.0.1:37622)
0619 04:38:05.041705 (+ 11122us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:05.041725 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:05.041858 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:05.042888 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:05.042896 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:05.042927 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:05.043327 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:05.043341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.045270 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.045274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:05.047689 (+ 2415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:05.047704 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.049184 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.049190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:05.049367 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:05.049420 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 04:38:05.049531 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 04:38:05.049716 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10984,"mutex_wait_us":22,"thread_start_us":893,"threads_started":1}
I20260619 04:38:05.050661 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:05.030369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32993 (local address 127.0.0.1:52500)
0619 04:38:05.031683 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:05.031725 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:05.031892 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:05.043809 (+ 11917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:05.043814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:05.043836 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:05.044222 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:05.044236 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.045904 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.045909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:05.047975 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:05.047992 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.049800 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.049806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:05.049980 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:05.050018 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 04:38:05.050077 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 04:38:05.050232 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"mutex_wait_us":31,"thread_start_us":11047,"threads_started":1}
I20260619 04:38:05.051630 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:05.030892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52500 (local address 127.0.97.2:32993)
0619 04:38:05.043274 (+ 12382us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:05.043283 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:05.043312 (+ 29us) server_negotiation.cc:407] Connection header received
0619 04:38:05.043405 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:05.043412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:05.043636 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:05.043805 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:05.044423 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.045754 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.048210 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.049582 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.050901 (+ 1319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:05.051087 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:05.051152 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 04:38:05.051327 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12200,"thread_start_us":13071,"threads_started":1}
I20260619 04:38:05.054612 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:05.033186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46113 (local address 127.0.0.1:49802)
0619 04:38:05.045053 (+ 11867us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:05.045072 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:05.045184 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:05.045745 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:05.045749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:05.045773 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:05.048236 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:05.048249 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.050104 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.050108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:05.052207 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:05.052222 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.053869 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.053875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:05.054034 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:05.054066 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 04:38:05.054163 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 04:38:05.054282 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11752,"mutex_wait_us":54,"thread_start_us":11624,"threads_started":1}
I20260619 04:38:05.054648 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:05.033934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49802 (local address 127.0.97.3:46113)
0619 04:38:05.044650 (+ 10716us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:05.044658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:05.045173 (+ 515us) server_negotiation.cc:407] Connection header received
0619 04:38:05.045319 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:05.045327 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:05.045556 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:05.045732 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:05.048408 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.049947 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.052385 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.053681 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.053802 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:05.054013 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:05.054203 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 04:38:05.054348 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10550,"thread_start_us":10368,"threads_started":1}
I20260619 04:38:05.054216 18766 tablet_service.cc:1467] Processing CreateTablet for tablet a780c2a5285c4317a7a6d7c974f7234a (DEFAULT_TABLE table=test-table [id=d05e2bcba2bc4262b136ca501e354d17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:38:05.055656 18766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a780c2a5285c4317a7a6d7c974f7234a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:05.055171 18900 tablet_service.cc:1467] Processing CreateTablet for tablet a780c2a5285c4317a7a6d7c974f7234a (DEFAULT_TABLE table=test-table [id=d05e2bcba2bc4262b136ca501e354d17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:38:05.056398 18900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a780c2a5285c4317a7a6d7c974f7234a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:05.058262 19033 tablet_service.cc:1467] Processing CreateTablet for tablet a780c2a5285c4317a7a6d7c974f7234a (DEFAULT_TABLE table=test-table [id=d05e2bcba2bc4262b136ca501e354d17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:38:05.059535 19033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a780c2a5285c4317a7a6d7c974f7234a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:05.066694 19116 tablet_bootstrap.cc:492] T a780c2a5285c4317a7a6d7c974f7234a P 8fd86f2f9f6d4423868542da14492333: Bootstrap starting.
I20260619 04:38:05.070364 19116 tablet_bootstrap.cc:654] T a780c2a5285c4317a7a6d7c974f7234a P 8fd86f2f9f6d4423868542da14492333: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:05.071127 19117 tablet_bootstrap.cc:492] T a780c2a5285c4317a7a6d7c974f7234a P bd1c51fa48e84198a35c39dad2257acc: Bootstrap starting.
I20260619 04:38:05.071614 19116 log.cc:824] T a780c2a5285c4317a7a6d7c974f7234a P 8fd86f2f9f6d4423868542da14492333: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:05.074496 19117 tablet_bootstrap.cc:654] T a780c2a5285c4317a7a6d7c974f7234a P bd1c51fa48e84198a35c39dad2257acc: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:05.074597 19116 tablet_bootstrap.cc:492] T a780c2a5285c4317a7a6d7c974f7234a P 8fd86f2f9f6d4423868542da14492333: No bootstrap required, opened a new log
I20260619 04:38:05.074858 19116 ts_tablet_manager.cc:1397] T a780c2a5285c4317a7a6d7c974f7234a P 8fd86f2f9f6d4423868542da14492333: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 04:38:05.075644 19117 log.cc:824] T a780c2a5285c4317a7a6d7c974f7234a P bd1c51fa48e84198a35c39dad2257acc: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:05.076576 19119 tablet_bootstrap.cc:492] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec: Bootstrap starting.
I20260619 04:38:05.078024 19117 tablet_bootstrap.cc:492] T a780c2a5285c4317a7a6d7c974f7234a P bd1c51fa48e84198a35c39dad2257acc: No bootstrap required, opened a new log
I20260619 04:38:05.078239 19117 ts_tablet_manager.cc:1397] T a780c2a5285c4317a7a6d7c974f7234a P bd1c51fa48e84198a35c39dad2257acc: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260619 04:38:05.079752 19119 tablet_bootstrap.cc:654] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:05.080924 19119 log.cc:824] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:05.083683 19119 tablet_bootstrap.cc:492] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec: No bootstrap required, opened a new log
I20260619 04:38:05.083926 19119 ts_tablet_manager.cc:1397] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 04:38:05.084597 19116 raft_consensus.cc:348] T a780c2a5285c4317a7a6d7c974f7234a P 8fd86f2f9f6d4423868542da14492333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd86f2f9f6d4423868542da14492333" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32993 } } peers { permanent_uuid: "bd1c51fa48e84198a35c39dad2257acc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "07d4476fb4484bada3b074fcb67ed3ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46113 } }
I20260619 04:38:05.085047 19116 raft_consensus.cc:374] T a780c2a5285c4317a7a6d7c974f7234a P 8fd86f2f9f6d4423868542da14492333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:05.085150 19116 raft_consensus.cc:729] T a780c2a5285c4317a7a6d7c974f7234a P 8fd86f2f9f6d4423868542da14492333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd86f2f9f6d4423868542da14492333, State: Initialized, Role: FOLLOWER
I20260619 04:38:05.085709 19117 raft_consensus.cc:348] T a780c2a5285c4317a7a6d7c974f7234a P bd1c51fa48e84198a35c39dad2257acc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd86f2f9f6d4423868542da14492333" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32993 } } peers { permanent_uuid: "bd1c51fa48e84198a35c39dad2257acc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "07d4476fb4484bada3b074fcb67ed3ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46113 } }
I20260619 04:38:05.085763 19116 consensus_queue.cc:260] T a780c2a5285c4317a7a6d7c974f7234a P 8fd86f2f9f6d4423868542da14492333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd86f2f9f6d4423868542da14492333" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32993 } } peers { permanent_uuid: "bd1c51fa48e84198a35c39dad2257acc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "07d4476fb4484bada3b074fcb67ed3ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46113 } }
I20260619 04:38:05.086015 19117 raft_consensus.cc:374] T a780c2a5285c4317a7a6d7c974f7234a P bd1c51fa48e84198a35c39dad2257acc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:05.086114 19117 raft_consensus.cc:729] T a780c2a5285c4317a7a6d7c974f7234a P bd1c51fa48e84198a35c39dad2257acc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1c51fa48e84198a35c39dad2257acc, State: Initialized, Role: FOLLOWER
I20260619 04:38:05.086778 18965 heartbeater.cc:503] Master 127.0.97.62:43829 was elected leader, sending a full tablet report...
I20260619 04:38:05.087424 19116 ts_tablet_manager.cc:1428] T a780c2a5285c4317a7a6d7c974f7234a P 8fd86f2f9f6d4423868542da14492333: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260619 04:38:05.087917 19117 consensus_queue.cc:260] T a780c2a5285c4317a7a6d7c974f7234a P bd1c51fa48e84198a35c39dad2257acc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd86f2f9f6d4423868542da14492333" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32993 } } peers { permanent_uuid: "bd1c51fa48e84198a35c39dad2257acc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "07d4476fb4484bada3b074fcb67ed3ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46113 } }
I20260619 04:38:05.089447 19117 ts_tablet_manager.cc:1428] T a780c2a5285c4317a7a6d7c974f7234a P bd1c51fa48e84198a35c39dad2257acc: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260619 04:38:05.093086 19119 raft_consensus.cc:348] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd86f2f9f6d4423868542da14492333" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32993 } } peers { permanent_uuid: "bd1c51fa48e84198a35c39dad2257acc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "07d4476fb4484bada3b074fcb67ed3ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46113 } }
I20260619 04:38:05.093482 19119 raft_consensus.cc:374] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:05.093575 19119 raft_consensus.cc:729] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07d4476fb4484bada3b074fcb67ed3ec, State: Initialized, Role: FOLLOWER
I20260619 04:38:05.093986 19119 consensus_queue.cc:260] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd86f2f9f6d4423868542da14492333" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32993 } } peers { permanent_uuid: "bd1c51fa48e84198a35c39dad2257acc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "07d4476fb4484bada3b074fcb67ed3ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46113 } }
I20260619 04:38:05.094816 19098 heartbeater.cc:503] Master 127.0.97.62:43829 was elected leader, sending a full tablet report...
I20260619 04:38:05.095374 19119 ts_tablet_manager.cc:1428] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260619 04:38:05.187099 19124 raft_consensus.cc:482] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:38:05.187383 19124 raft_consensus.cc:504] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd86f2f9f6d4423868542da14492333" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32993 } } peers { permanent_uuid: "bd1c51fa48e84198a35c39dad2257acc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "07d4476fb4484bada3b074fcb67ed3ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46113 } }
I20260619 04:38:05.189061 19124 leader_election.cc:288] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fd86f2f9f6d4423868542da14492333 (127.0.97.2:32993), bd1c51fa48e84198a35c39dad2257acc (127.0.97.1:32987)
I20260619 04:38:05.195221 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:05.189557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39543 (local address 127.0.97.2:32993)
0619 04:38:05.189755 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:05.189761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:05.189783 (+ 22us) server_negotiation.cc:407] Connection header received
0619 04:38:05.190044 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:05.190049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:05.190131 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:05.190234 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:05.190615 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.191537 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.193390 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.194460 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.194558 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:05.194626 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:05.194944 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 04:38:05.195031 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:38:05.195315 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:05.189374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32993 (local address 127.0.97.3:39543)
0619 04:38:05.189712 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:05.189735 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:05.189893 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:05.190275 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:05.190278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:05.190301 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:05.190475 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:05.190486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.191669 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.191675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:05.193233 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:05.193245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.194599 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.194605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:05.194786 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:05.194820 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 04:38:05.194928 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 04:38:05.195057 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20260619 04:38:05.196116 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:05.189557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40523 (local address 127.0.97.1:32987)
0619 04:38:05.189755 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:05.189761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:05.190379 (+ 618us) server_negotiation.cc:407] Connection header received
0619 04:38:05.190566 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:05.190570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:05.190640 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:05.190712 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:05.191259 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.192186 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.194175 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.195210 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.195333 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:05.195438 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:05.195851 (+ 413us) server_negotiation.cc:299] Negotiation successful
0619 04:38:05.195952 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 04:38:05.196260 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:05.189420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32987 (local address 127.0.97.3:40523)
0619 04:38:05.190262 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:05.190278 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:05.190435 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:05.190742 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:05.190746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:05.190770 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:05.191096 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:05.191108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.192357 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.192361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:05.194033 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:05.194048 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.195559 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.195564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:05.195685 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:05.195716 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 04:38:05.195797 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 04:38:05.195910 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":1,"spinlock_wait_cycles":20736,"thread_start_us":471,"threads_started":1}
I20260619 04:38:05.196378 18920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a780c2a5285c4317a7a6d7c974f7234a" candidate_uuid: "07d4476fb4484bada3b074fcb67ed3ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd86f2f9f6d4423868542da14492333" is_pre_election: true
I20260619 04:38:05.196897 18920 raft_consensus.cc:2454] T a780c2a5285c4317a7a6d7c974f7234a P 8fd86f2f9f6d4423868542da14492333 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07d4476fb4484bada3b074fcb67ed3ec in term 0.
I20260619 04:38:05.197296 18786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a780c2a5285c4317a7a6d7c974f7234a" candidate_uuid: "07d4476fb4484bada3b074fcb67ed3ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd1c51fa48e84198a35c39dad2257acc" is_pre_election: true
I20260619 04:38:05.197539 18986 leader_election.cc:302] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07d4476fb4484bada3b074fcb67ed3ec, 8fd86f2f9f6d4423868542da14492333; no voters:
I20260619 04:38:05.197933 19124 raft_consensus.cc:2781] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:38:05.197909 18786 raft_consensus.cc:2454] T a780c2a5285c4317a7a6d7c974f7234a P bd1c51fa48e84198a35c39dad2257acc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07d4476fb4484bada3b074fcb67ed3ec in term 0.
I20260619 04:38:05.198060 19124 raft_consensus.cc:482] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:38:05.198159 19124 raft_consensus.cc:3037] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:05.200373 19124 raft_consensus.cc:504] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd86f2f9f6d4423868542da14492333" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32993 } } peers { permanent_uuid: "bd1c51fa48e84198a35c39dad2257acc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "07d4476fb4484bada3b074fcb67ed3ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46113 } }
I20260619 04:38:05.200984 19124 leader_election.cc:288] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec [CANDIDATE]: Term 1 election: Requested vote from peers 8fd86f2f9f6d4423868542da14492333 (127.0.97.2:32993), bd1c51fa48e84198a35c39dad2257acc (127.0.97.1:32987)
I20260619 04:38:05.201382 18786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a780c2a5285c4317a7a6d7c974f7234a" candidate_uuid: "07d4476fb4484bada3b074fcb67ed3ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd1c51fa48e84198a35c39dad2257acc"
I20260619 04:38:05.201388 18920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a780c2a5285c4317a7a6d7c974f7234a" candidate_uuid: "07d4476fb4484bada3b074fcb67ed3ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd86f2f9f6d4423868542da14492333"
I20260619 04:38:05.201545 18920 raft_consensus.cc:3037] T a780c2a5285c4317a7a6d7c974f7234a P 8fd86f2f9f6d4423868542da14492333 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:05.201555 18786 raft_consensus.cc:3037] T a780c2a5285c4317a7a6d7c974f7234a P bd1c51fa48e84198a35c39dad2257acc [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:05.203621 18920 raft_consensus.cc:2454] T a780c2a5285c4317a7a6d7c974f7234a P 8fd86f2f9f6d4423868542da14492333 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07d4476fb4484bada3b074fcb67ed3ec in term 1.
I20260619 04:38:05.203630 18786 raft_consensus.cc:2454] T a780c2a5285c4317a7a6d7c974f7234a P bd1c51fa48e84198a35c39dad2257acc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07d4476fb4484bada3b074fcb67ed3ec in term 1.
I20260619 04:38:05.203984 18986 leader_election.cc:302] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07d4476fb4484bada3b074fcb67ed3ec, 8fd86f2f9f6d4423868542da14492333; no voters:
I20260619 04:38:05.204238 19124 raft_consensus.cc:2781] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:05.204696 19124 raft_consensus.cc:686] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec [term 1 LEADER]: Becoming Leader. State: Replica: 07d4476fb4484bada3b074fcb67ed3ec, State: Running, Role: LEADER
I20260619 04:38:05.205127 19124 consensus_queue.cc:237] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd86f2f9f6d4423868542da14492333" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32993 } } peers { permanent_uuid: "bd1c51fa48e84198a35c39dad2257acc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "07d4476fb4484bada3b074fcb67ed3ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46113 } }
W20260619 04:38:05.209580 19099 tablet.cc:2369] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:38:05.209815 18642 catalog_manager.cc:5495] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 07d4476fb4484bada3b074fcb67ed3ec (127.0.97.3). New cstate: current_term: 1 leader_uuid: "07d4476fb4484bada3b074fcb67ed3ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd86f2f9f6d4423868542da14492333" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd1c51fa48e84198a35c39dad2257acc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07d4476fb4484bada3b074fcb67ed3ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46113 } health_report { overall_health: HEALTHY } } }
W20260619 04:38:05.227998 18966 tablet.cc:2369] T a780c2a5285c4317a7a6d7c974f7234a P 8fd86f2f9f6d4423868542da14492333: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:38:05.232487 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:05.227136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49810 (local address 127.0.97.3:46113)
0619 04:38:05.227299 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:05.227304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:05.227321 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:38:05.227421 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:05.227425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:05.227506 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:05.227627 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:05.228040 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.228864 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.230391 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.230864 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.230985 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:05.231490 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:05.231986 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:05.232108 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:05.232218 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 04:38:05.232258 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 04:38:05.233505 18832 tablet.cc:2369] T a780c2a5285c4317a7a6d7c974f7234a P bd1c51fa48e84198a35c39dad2257acc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:38:05.240620 18786 raft_consensus.cc:1261] T a780c2a5285c4317a7a6d7c974f7234a P bd1c51fa48e84198a35c39dad2257acc [term 1 FOLLOWER]: Refusing update from remote peer 07d4476fb4484bada3b074fcb67ed3ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:38:05.240620 18920 raft_consensus.cc:1261] T a780c2a5285c4317a7a6d7c974f7234a P 8fd86f2f9f6d4423868542da14492333 [term 1 FOLLOWER]: Refusing update from remote peer 07d4476fb4484bada3b074fcb67ed3ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:38:05.241663 19126 consensus_queue.cc:1035] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fd86f2f9f6d4423868542da14492333" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:38:05.242172 19124 consensus_queue.cc:1035] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd1c51fa48e84198a35c39dad2257acc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:38:05.253927 19131 mvcc.cc:205] Tried to move back new op lower bound from 7298432553935572992 to 7298432553803698176. Current Snapshot: MvccSnapshot[applied={T|T < 7298432553935572992}]
I20260619 04:38:05.255501 19129 mvcc.cc:205] Tried to move back new op lower bound from 7298432553935572992 to 7298432553803698176. Current Snapshot: MvccSnapshot[applied={T|T < 7298432553935572992 or (T in {7298432553935572992})}]
I20260619 04:38:05.265219 18642 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:38:05.265466 18642 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:38:05.268639 18642 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f85d698695f4e87b6da8f6b1092e682: Sending DeleteTablet for 3 replicas of tablet a780c2a5285c4317a7a6d7c974f7234a
I20260619 04:38:05.269761 18900 tablet_service.cc:1514] Processing DeleteTablet for tablet a780c2a5285c4317a7a6d7c974f7234a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:38:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52500
I20260619 04:38:05.269764 18766 tablet_service.cc:1514] Processing DeleteTablet for tablet a780c2a5285c4317a7a6d7c974f7234a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:38:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37622
I20260619 04:38:05.269824 19033 tablet_service.cc:1514] Processing DeleteTablet for tablet a780c2a5285c4317a7a6d7c974f7234a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:38:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49802
I20260619 04:38:05.273252 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 18703
I20260619 04:38:05.282320 19140 tablet_replica.cc:330] T a780c2a5285c4317a7a6d7c974f7234a P 8fd86f2f9f6d4423868542da14492333: stopping tablet replica
I20260619 04:38:05.282986 19140 raft_consensus.cc:2229] T a780c2a5285c4317a7a6d7c974f7234a P 8fd86f2f9f6d4423868542da14492333 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:38:05.283582 19140 raft_consensus.cc:2258] T a780c2a5285c4317a7a6d7c974f7234a P 8fd86f2f9f6d4423868542da14492333 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:38:05.285981 19140 ts_tablet_manager.cc:1905] T a780c2a5285c4317a7a6d7c974f7234a P 8fd86f2f9f6d4423868542da14492333: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:38:05.286324 19139 tablet_replica.cc:330] T a780c2a5285c4317a7a6d7c974f7234a P bd1c51fa48e84198a35c39dad2257acc: stopping tablet replica
I20260619 04:38:05.286821 19139 raft_consensus.cc:2229] T a780c2a5285c4317a7a6d7c974f7234a P bd1c51fa48e84198a35c39dad2257acc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:38:05.288038 19141 tablet_replica.cc:330] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec: stopping tablet replica
I20260619 04:38:05.288228 19139 raft_consensus.cc:2258] T a780c2a5285c4317a7a6d7c974f7234a P bd1c51fa48e84198a35c39dad2257acc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:38:05.288508 19141 raft_consensus.cc:2229] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:38:05.289165 19141 raft_consensus.cc:2258] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:38:05.291486 19141 ts_tablet_manager.cc:1905] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 04:38:05 dist-test-slave-n3tq krb5kdc[18597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 04:38:05.293411 19139 ts_tablet_manager.cc:1905] T a780c2a5285c4317a7a6d7c974f7234a P bd1c51fa48e84198a35c39dad2257acc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:38:05.298579 19141 ts_tablet_manager.cc:1918] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:38:05.298841 19141 log.cc:1196] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/a780c2a5285c4317a7a6d7c974f7234a
I20260619 04:38:05.299415 19141 ts_tablet_manager.cc:1939] T a780c2a5285c4317a7a6d7c974f7234a P 07d4476fb4484bada3b074fcb67ed3ec: Deleting consensus metadata
I20260619 04:38:05.300956 19139 ts_tablet_manager.cc:1918] T a780c2a5285c4317a7a6d7c974f7234a P bd1c51fa48e84198a35c39dad2257acc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:38:05.301189 19139 log.cc:1196] T a780c2a5285c4317a7a6d7c974f7234a P bd1c51fa48e84198a35c39dad2257acc: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-0/wal/wals/a780c2a5285c4317a7a6d7c974f7234a
I20260619 04:38:05.301484 18628 catalog_manager.cc:4841] TS 07d4476fb4484bada3b074fcb67ed3ec (127.0.97.3:46113): tablet a780c2a5285c4317a7a6d7c974f7234a (table test-table [id=d05e2bcba2bc4262b136ca501e354d17]) successfully deleted
I20260619 04:38:05.301741 19139 ts_tablet_manager.cc:1939] T a780c2a5285c4317a7a6d7c974f7234a P bd1c51fa48e84198a35c39dad2257acc: Deleting consensus metadata
I20260619 04:38:05.303139 18627 catalog_manager.cc:4841] TS bd1c51fa48e84198a35c39dad2257acc (127.0.97.1:32987): tablet a780c2a5285c4317a7a6d7c974f7234a (table test-table [id=d05e2bcba2bc4262b136ca501e354d17]) successfully deleted
I20260619 04:38:05.303282 19140 ts_tablet_manager.cc:1918] T a780c2a5285c4317a7a6d7c974f7234a P 8fd86f2f9f6d4423868542da14492333: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:38:05.303452 19140 log.cc:1196] T a780c2a5285c4317a7a6d7c974f7234a P 8fd86f2f9f6d4423868542da14492333: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/a780c2a5285c4317a7a6d7c974f7234a
I20260619 04:38:05.303972 19140 ts_tablet_manager.cc:1939] T a780c2a5285c4317a7a6d7c974f7234a P 8fd86f2f9f6d4423868542da14492333: Deleting consensus metadata
I20260619 04:38:05.304209 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:05.275623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37626 (local address 127.0.97.1:32987)
0619 04:38:05.275797 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:05.275803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:05.278404 (+ 2601us) server_negotiation.cc:407] Connection header received
0619 04:38:05.279255 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:05.279262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:05.279365 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:05.279438 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:05.283055 (+ 3617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.284132 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.286044 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.286674 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.286775 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:05.292827 (+ 6052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:05.292854 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:05.292872 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:05.292924 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:05.296776 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:05.297651 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:05.297661 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:05.297687 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:05.297858 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:05.298317 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:05.298322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:05.298325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:05.298821 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:05.299562 (+ 741us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:05.303593 (+ 4031us) server_negotiation.cc:299] Negotiation successful
0619 04:38:05.303856 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:38:05.305536 18826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:05.305786 18626 catalog_manager.cc:4841] TS 8fd86f2f9f6d4423868542da14492333 (127.0.97.2:32993): tablet a780c2a5285c4317a7a6d7c974f7234a (table test-table [id=d05e2bcba2bc4262b136ca501e354d17]) successfully deleted
I20260619 04:38:05.405458 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 18703
I20260619 04:38:05.420253 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 18837
Jun 19 04:38:05 dist-test-slave-n3tq krb5kdc[18597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 04:38:05.435199 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:05.421051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52502 (local address 127.0.97.2:32993)
0619 04:38:05.421243 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:05.421249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:05.421277 (+ 28us) server_negotiation.cc:407] Connection header received
0619 04:38:05.421342 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:05.421348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:05.421424 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:05.421506 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:05.422707 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.423488 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.424761 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.425151 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.425264 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:05.429676 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:05.429691 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:05.429705 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:05.429766 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:05.432642 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:05.433278 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:05.433287 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:05.433291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:05.433409 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:05.433748 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:05.433753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:05.433756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:05.434112 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:05.434402 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:05.434674 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 04:38:05.434873 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 04:38:05.435837 18960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:05.579903 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 18837
I20260619 04:38:05.594696 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 18970
Jun 19 04:38:05 dist-test-slave-n3tq krb5kdc[18597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 04:38:05.610558 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:05.595414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49826 (local address 127.0.97.3:46113)
0619 04:38:05.595601 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:05.595606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:05.595712 (+ 106us) server_negotiation.cc:407] Connection header received
0619 04:38:05.595936 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:05.595943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:05.596024 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:05.596126 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:05.597421 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.598214 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.599469 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:05.599904 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:05.600013 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:05.604536 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:05.604549 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:05.604564 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:05.604599 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:05.607791 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:05.608493 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:05.608503 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:05.608507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:05.608650 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:05.609054 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:05.609059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:05.609062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:05.609401 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:05.609714 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:05.609974 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 04:38:05.610182 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 04:38:05.611290 19093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:05.731276 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 18970
I20260619 04:38:05.746877 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 18610
I20260619 04:38:05.747941 18672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:05.858083 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 18610
2026-06-19T04:38:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4128 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T04:38:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:05Z Disabled control of system clock
I20260619 04:38:05.927102 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35135
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 04:38:06.245893 19150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:06.246183 19150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:06.246286 19150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:06.253618 19150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:06.253736 19150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:06.253779 19150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:06.253857 19150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:06.253919 19150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:06.263182 19150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35135
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:06.264437 19150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:06.266201 19150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:06.277995 19156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:06.278064 19155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:06.278098 19158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:06.278647 19150 server_base.cc:1034] running on GCE node
I20260619 04:38:06.279450 19150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:06.280885 19150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:06.282080 19150 hybrid_clock.cc:648] HybridClock initialized: now 1781843886282020 us; error 50 us; skew 500 ppm
I20260619 04:38:06.282398 19150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:38:06.284592 19150 webserver.cc:460] Webserver started at http://127.0.97.62:41073/ using document root <none> and password file <none>
I20260619 04:38:06.285254 19150 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:06.285341 19150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:06.285604 19150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:06.288568 19150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "dbc3f4cf6e9249d6b39538a0fb9af190"
format_stamp: "Formatted at 2026-06-19 04:38:06 on dist-test-slave-n3tq"
server_key: "fea9a78a036305bd40efbfe11dc8d661"
server_key_iv: "ffe28beccf6dde83245f0b23d8e984c7"
server_key_version: "encryptionkey@0"
I20260619 04:38:06.289338 19150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "dbc3f4cf6e9249d6b39538a0fb9af190"
format_stamp: "Formatted at 2026-06-19 04:38:06 on dist-test-slave-n3tq"
server_key: "fea9a78a036305bd40efbfe11dc8d661"
server_key_iv: "ffe28beccf6dde83245f0b23d8e984c7"
server_key_version: "encryptionkey@0"
I20260619 04:38:06.294266 19150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260619 04:38:06.297958 19164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:06.299300 19150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:38:06.299458 19150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "dbc3f4cf6e9249d6b39538a0fb9af190"
format_stamp: "Formatted at 2026-06-19 04:38:06 on dist-test-slave-n3tq"
server_key: "fea9a78a036305bd40efbfe11dc8d661"
server_key_iv: "ffe28beccf6dde83245f0b23d8e984c7"
server_key_version: "encryptionkey@0"
I20260619 04:38:06.299647 19150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:06.317322 19150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:06.318167 19150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:06.318419 19150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:06.335815 19150 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35135
I20260619 04:38:06.335834 19215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35135 every 8 connection(s)
I20260619 04:38:06.337138 19150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:06.339540 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 19150
I20260619 04:38:06.339725 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:06.340281 388 external_mini_cluster.cc:1428] Setting key d4838da029492f976ac595cb37e2fc4b
I20260619 04:38:06.341450 19216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:06.353487 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:06.342256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36118 (local address 127.0.97.62:35135)
0619 04:38:06.343266 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:06.343276 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:06.343318 (+ 42us) server_negotiation.cc:407] Connection header received
0619 04:38:06.344928 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:06.344953 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:06.345483 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:06.345883 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:06.346457 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:06.347801 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:06.349208 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:06.349878 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:06.350035 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:06.351200 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:06.351211 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:06.351226 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:06.351808 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:06.351886 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:06.351907 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:06.352006 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:06.352213 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 04:38:06.352546 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":375,"threads_started":1}
I20260619 04:38:06.353837 19216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190: Bootstrap starting.
I20260619 04:38:06.357165 19216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:06.358107 19216 log.cc:824] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:06.360739 19216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190: No bootstrap required, opened a new log
I20260619 04:38:06.367888 19216 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc3f4cf6e9249d6b39538a0fb9af190" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35135 } }
I20260619 04:38:06.368238 19216 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:06.368317 19216 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc3f4cf6e9249d6b39538a0fb9af190, State: Initialized, Role: FOLLOWER
I20260619 04:38:06.368848 19216 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc3f4cf6e9249d6b39538a0fb9af190" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35135 } }
I20260619 04:38:06.369063 19216 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:38:06.369192 19216 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:38:06.369372 19216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:06.371511 19216 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc3f4cf6e9249d6b39538a0fb9af190" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35135 } }
I20260619 04:38:06.371989 19216 leader_election.cc:302] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbc3f4cf6e9249d6b39538a0fb9af190; no voters:
I20260619 04:38:06.372530 19216 leader_election.cc:288] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:38:06.372721 19221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:06.373356 19221 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190 [term 1 LEADER]: Becoming Leader. State: Replica: dbc3f4cf6e9249d6b39538a0fb9af190, State: Running, Role: LEADER
I20260619 04:38:06.373842 19221 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc3f4cf6e9249d6b39538a0fb9af190" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35135 } }
I20260619 04:38:06.374423 19216 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:06.376560 19223 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbc3f4cf6e9249d6b39538a0fb9af190. Latest consensus state: current_term: 1 leader_uuid: "dbc3f4cf6e9249d6b39538a0fb9af190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc3f4cf6e9249d6b39538a0fb9af190" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35135 } } }
I20260619 04:38:06.376986 19223 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:06.380836 19222 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbc3f4cf6e9249d6b39538a0fb9af190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc3f4cf6e9249d6b39538a0fb9af190" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35135 } } }
I20260619 04:38:06.381011 19230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:06.381067 19222 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:06.386448 19230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:06.395771 19230 catalog_manager.cc:1347] Generated new cluster ID: f166533ee624459fa579e416b6b11152
I20260619 04:38:06.395884 19230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:06.404155 19230 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:06.406049 19230 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:06.419070 19230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190: Generated new TSK 0
I20260619 04:38:06.420202 19230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:38:06.425563 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35135
--builtin_ntp_servers=127.0.97.20:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 04:38:06.762287 19240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:06.762573 19240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:06.762656 19240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:06.770073 19240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:06.770179 19240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:06.770298 19240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:38:06.780732 19240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:06.781957 19240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:06.783738 19240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:06.796286 19248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:06.796362 19246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:06.796536 19245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:06.797420 19240 server_base.cc:1034] running on GCE node
I20260619 04:38:06.797993 19240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:06.798919 19240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:06.800123 19240 hybrid_clock.cc:648] HybridClock initialized: now 1781843886800084 us; error 40 us; skew 500 ppm
I20260619 04:38:06.800419 19240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:38:06.802598 19240 webserver.cc:460] Webserver started at http://127.0.97.1:43217/ using document root <none> and password file <none>
I20260619 04:38:06.803259 19240 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:06.803354 19240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:06.803658 19240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:06.806600 19240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "33559e87a0404e0c8ea4c695dfe07427"
format_stamp: "Formatted at 2026-06-19 04:38:06 on dist-test-slave-n3tq"
server_key: "434f3ad7e8e6afcaab328811ef5a2e47"
server_key_iv: "e58f0c6bc1e4909d89d1e051e7c797e1"
server_key_version: "encryptionkey@0"
I20260619 04:38:06.807365 19240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "33559e87a0404e0c8ea4c695dfe07427"
format_stamp: "Formatted at 2026-06-19 04:38:06 on dist-test-slave-n3tq"
server_key: "434f3ad7e8e6afcaab328811ef5a2e47"
server_key_iv: "e58f0c6bc1e4909d89d1e051e7c797e1"
server_key_version: "encryptionkey@0"
I20260619 04:38:06.812124 19240 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260619 04:38:06.815485 19254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:06.816792 19240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 04:38:06.816936 19240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "33559e87a0404e0c8ea4c695dfe07427"
format_stamp: "Formatted at 2026-06-19 04:38:06 on dist-test-slave-n3tq"
server_key: "434f3ad7e8e6afcaab328811ef5a2e47"
server_key_iv: "e58f0c6bc1e4909d89d1e051e7c797e1"
server_key_version: "encryptionkey@0"
I20260619 04:38:06.817150 19240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:06.829037 19240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:06.829861 19240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:06.830116 19240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:06.831080 19240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:38:06.832650 19240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:38:06.832746 19240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:06.832856 19240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:38:06.832923 19240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:06.863684 19240 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39311
I20260619 04:38:06.863737 19366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39311 every 8 connection(s)
I20260619 04:38:06.865046 19240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:38:06.868170 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 19240
I20260619 04:38:06.868330 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:38:06.868878 388 external_mini_cluster.cc:1428] Setting key 696510fdc2cc85e08118a23bc570046d
I20260619 04:38:06.871726 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35135
--builtin_ntp_servers=127.0.97.20:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 04:38:06.878553 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:06.867527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39593 (local address 127.0.97.62:35135)
0619 04:38:06.868055 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:06.868060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:06.869411 (+ 1351us) server_negotiation.cc:407] Connection header received
0619 04:38:06.871337 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:06.871342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:06.871407 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:06.871493 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:06.872873 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:06.873625 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:06.875593 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:06.876005 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:06.876105 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:06.877614 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:06.877620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:06.877623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:06.877642 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:06.877691 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:06.877704 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:06.877817 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:06.878087 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 04:38:06.878185 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":257,"threads_started":1}
I20260619 04:38:06.879652 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:06.868115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35135 (local address 127.0.97.1:39593)
0619 04:38:06.869229 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:06.869286 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:06.871040 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:06.871773 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:06.871786 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:06.871919 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:06.872699 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:06.872725 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:06.873773 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:06.873777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:06.875457 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:06.875472 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:06.876206 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:06.876212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:06.876495 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:06.877256 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:38:06.877260 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:06.877315 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:38:06.877410 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:06.877843 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:06.877860 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:06.877867 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 04:38:06.878039 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 04:38:06.878440 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":343,"threads_started":1}
I20260619 04:38:06.881242 19367 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35135
I20260619 04:38:06.881520 19367 heartbeater.cc:464] Registering TS with master...
I20260619 04:38:06.882238 19367 heartbeater.cc:511] Master 127.0.97.62:35135 requested a full tablet report, sending...
I20260619 04:38:06.884194 19181 ts_manager.cc:194] Registered new tserver with Master: 33559e87a0404e0c8ea4c695dfe07427 (127.0.97.1:39311)
I20260619 04:38:06.885773 19181 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:39593
W20260619 04:38:07.194303 19371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:07.194602 19371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:07.194737 19371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:07.202059 19371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:07.202171 19371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:07.202313 19371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:38:07.211611 19371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:07.212967 19371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:07.214749 19371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:07.227409 19376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:07.227344 19377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:07.227378 19379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:07.228125 19371 server_base.cc:1034] running on GCE node
I20260619 04:38:07.228812 19371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:07.229782 19371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:07.230983 19371 hybrid_clock.cc:648] HybridClock initialized: now 1781843887230932 us; error 47 us; skew 500 ppm
I20260619 04:38:07.231288 19371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:38:07.233539 19371 webserver.cc:460] Webserver started at http://127.0.97.2:34369/ using document root <none> and password file <none>
I20260619 04:38:07.234231 19371 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:07.234330 19371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:07.234617 19371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:07.237489 19371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "82896a0ad07a445ebef24c9425b797bb"
format_stamp: "Formatted at 2026-06-19 04:38:07 on dist-test-slave-n3tq"
server_key: "f0401b0fa70bc5f70154f164d0b4a255"
server_key_iv: "bf3453d53c0f5f420508f62cd350abf0"
server_key_version: "encryptionkey@0"
I20260619 04:38:07.238263 19371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "82896a0ad07a445ebef24c9425b797bb"
format_stamp: "Formatted at 2026-06-19 04:38:07 on dist-test-slave-n3tq"
server_key: "f0401b0fa70bc5f70154f164d0b4a255"
server_key_iv: "bf3453d53c0f5f420508f62cd350abf0"
server_key_version: "encryptionkey@0"
I20260619 04:38:07.243274 19371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 04:38:07.246753 19385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:07.248085 19371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:38:07.248236 19371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "82896a0ad07a445ebef24c9425b797bb"
format_stamp: "Formatted at 2026-06-19 04:38:07 on dist-test-slave-n3tq"
server_key: "f0401b0fa70bc5f70154f164d0b4a255"
server_key_iv: "bf3453d53c0f5f420508f62cd350abf0"
server_key_version: "encryptionkey@0"
I20260619 04:38:07.248445 19371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:07.273468 19371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:07.274333 19371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:07.274583 19371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:07.275532 19371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:38:07.277040 19371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:38:07.277153 19371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:07.277256 19371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:38:07.277328 19371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:07.307723 19371 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37939
I20260619 04:38:07.307813 19497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37939 every 8 connection(s)
I20260619 04:38:07.309007 19371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:38:07.316630 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 19371
I20260619 04:38:07.316800 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:38:07.317262 388 external_mini_cluster.cc:1428] Setting key da6a31258d21efdd2b7edb4efa9e887f
I20260619 04:38:07.319979 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35135
--builtin_ntp_servers=127.0.97.20:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 04:38:07.321592 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:07.311368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57613 (local address 127.0.97.62:35135)
0619 04:38:07.311529 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:07.311534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:07.312894 (+ 1360us) server_negotiation.cc:407] Connection header received
0619 04:38:07.314621 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:07.314626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:07.314700 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:07.314797 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:07.315956 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:07.316926 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:07.318759 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:07.319326 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:07.319443 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:07.320746 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:07.320753 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:07.320758 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:07.320779 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:07.320830 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:07.320844 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:07.320932 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:07.321110 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 04:38:07.321212 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 04:38:07.322588 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:07.311830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35135 (local address 127.0.97.2:57613)
0619 04:38:07.312738 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:07.312777 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:07.314345 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:07.315086 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:07.315098 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:07.315221 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:07.315782 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:07.315799 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:07.317077 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:07.317080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:07.318602 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:07.318615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:07.319537 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:07.319542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:07.319743 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:07.320398 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:38:07.320400 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:07.320445 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:38:07.320567 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:07.320933 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:07.320941 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:07.320947 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 04:38:07.321086 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 04:38:07.321445 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":271,"threads_started":1}
I20260619 04:38:07.323874 19498 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35135
I20260619 04:38:07.324149 19498 heartbeater.cc:464] Registering TS with master...
I20260619 04:38:07.324888 19498 heartbeater.cc:511] Master 127.0.97.62:35135 requested a full tablet report, sending...
I20260619 04:38:07.326287 19181 ts_manager.cc:194] Registered new tserver with Master: 82896a0ad07a445ebef24c9425b797bb (127.0.97.2:37939)
I20260619 04:38:07.327257 19181 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:57613
W20260619 04:38:07.634230 19501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:07.634526 19501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:07.634662 19501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:07.641901 19501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:07.642017 19501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:07.642163 19501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:38:07.651242 19501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:07.652539 19501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:07.654304 19501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:07.666736 19506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:07.666707 19507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:07.666757 19509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:07.667538 19501 server_base.cc:1034] running on GCE node
I20260619 04:38:07.668092 19501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:07.669023 19501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:07.670236 19501 hybrid_clock.cc:648] HybridClock initialized: now 1781843887670174 us; error 48 us; skew 500 ppm
I20260619 04:38:07.670531 19501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:38:07.672740 19501 webserver.cc:460] Webserver started at http://127.0.97.3:37287/ using document root <none> and password file <none>
I20260619 04:38:07.673381 19501 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:07.673475 19501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:07.673771 19501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:07.676577 19501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "6e59595e00e640a8b8ee64c509ef8447"
format_stamp: "Formatted at 2026-06-19 04:38:07 on dist-test-slave-n3tq"
server_key: "a7bab33a309dca2123ec9f850fb4af5f"
server_key_iv: "8a4f0cac25fd653210470f37549b61ec"
server_key_version: "encryptionkey@0"
I20260619 04:38:07.677326 19501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6e59595e00e640a8b8ee64c509ef8447"
format_stamp: "Formatted at 2026-06-19 04:38:07 on dist-test-slave-n3tq"
server_key: "a7bab33a309dca2123ec9f850fb4af5f"
server_key_iv: "8a4f0cac25fd653210470f37549b61ec"
server_key_version: "encryptionkey@0"
I20260619 04:38:07.682221 19501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 04:38:07.685668 19515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:07.687037 19501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:38:07.687204 19501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "6e59595e00e640a8b8ee64c509ef8447"
format_stamp: "Formatted at 2026-06-19 04:38:07 on dist-test-slave-n3tq"
server_key: "a7bab33a309dca2123ec9f850fb4af5f"
server_key_iv: "8a4f0cac25fd653210470f37549b61ec"
server_key_version: "encryptionkey@0"
I20260619 04:38:07.687405 19501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:07.702533 19501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:07.703344 19501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:07.703588 19501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:07.704486 19501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:38:07.705910 19501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:38:07.706004 19501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:07.706108 19501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:38:07.706179 19501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:07.736785 19501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44203
I20260619 04:38:07.736845 19627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44203 every 8 connection(s)
I20260619 04:38:07.738122 19501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:38:07.744024 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 19501
I20260619 04:38:07.744215 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:38:07.744750 388 external_mini_cluster.cc:1428] Setting key 8d9099101ab7e00b09c6b5af259e8575
I20260619 04:38:07.750689 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:07.740622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34413 (local address 127.0.97.62:35135)
0619 04:38:07.740807 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:07.740812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:07.742212 (+ 1400us) server_negotiation.cc:407] Connection header received
0619 04:38:07.744171 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:07.744178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:07.744267 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:07.744372 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:07.745818 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:07.746794 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:07.748319 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:07.748761 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:07.748852 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:07.750037 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:07.750050 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:07.750054 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:07.750071 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:07.750103 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:07.750115 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:07.750177 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:07.750333 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 04:38:07.750435 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 04:38:07.751652 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:07.741165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35135 (local address 127.0.97.3:34413)
0619 04:38:07.742043 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:07.742084 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:07.743810 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:07.744661 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:07.744673 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:07.744813 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:07.745582 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:07.745608 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:07.746923 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:07.746927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:07.748186 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:07.748198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:07.748929 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:07.748934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:07.749136 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:07.749761 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:38:07.749764 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:07.749804 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:38:07.749881 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:07.750175 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:07.750182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:07.750188 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 04:38:07.750319 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 04:38:07.750623 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":257,"threads_started":1}
I20260619 04:38:07.753036 19628 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35135
I20260619 04:38:07.753330 19628 heartbeater.cc:464] Registering TS with master...
I20260619 04:38:07.754051 19628 heartbeater.cc:511] Master 127.0.97.62:35135 requested a full tablet report, sending...
I20260619 04:38:07.755363 19181 ts_manager.cc:194] Registered new tserver with Master: 6e59595e00e640a8b8ee64c509ef8447 (127.0.97.3:44203)
I20260619 04:38:07.756137 19181 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:34413
I20260619 04:38:07.759722 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:38:07.768862 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:07.763551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36134 (local address 127.0.97.62:35135)
0619 04:38:07.763742 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:07.763749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:07.764068 (+ 319us) server_negotiation.cc:407] Connection header received
0619 04:38:07.764186 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:07.764190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:07.764250 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:07.764326 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:07.764809 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:07.765503 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:07.766766 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:07.767156 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:07.767244 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:07.768208 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:07.768212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:07.768215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:07.768232 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:07.768259 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:07.768269 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:07.768361 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:07.768513 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 04:38:07.768563 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:38:07.772411 19181 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:38:07.780728 19181 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:38:07.803820 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:07.792568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39311 (local address 127.0.0.1:49634)
0619 04:38:07.794637 (+ 2069us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:07.794654 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:07.794846 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:07.795816 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:07.795824 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:07.795856 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:07.796272 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:07.796284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:07.798160 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:07.798165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:07.800691 (+ 2526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:07.800709 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:07.802867 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:07.802874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:07.803079 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:07.803133 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 04:38:07.803273 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 04:38:07.803469 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1939,"threads_started":1}
I20260619 04:38:07.804807 19641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:07.792589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49634 (local address 127.0.97.1:39311)
0619 04:38:07.793993 (+ 1404us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:07.794001 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:07.794744 (+ 743us) server_negotiation.cc:407] Connection header received
0619 04:38:07.795033 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:07.795042 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:07.795328 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:07.795499 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:07.796497 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:07.798014 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:07.800894 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:07.802318 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:07.802423 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:07.802686 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:07.804249 (+ 1563us) server_negotiation.cc:299] Negotiation successful
0619 04:38:07.804444 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1247,"thread_start_us":1152,"threads_started":1}
I20260619 04:38:07.812345 19302 tablet_service.cc:1467] Processing CreateTablet for tablet 982e51c3f9084e519b85ccce4bb38644 (DEFAULT_TABLE table=test-table [id=f4d76baf491146a78bd3ec4786574c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:38:07.813875 19302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 982e51c3f9084e519b85ccce4bb38644. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:07.814620 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:07.792547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44203 (local address 127.0.0.1:59310)
0619 04:38:07.793215 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:07.793254 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:07.793457 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:07.808121 (+ 14664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:07.808126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:07.808149 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:07.808526 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:07.808540 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:07.810501 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:07.810505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:07.812427 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:07.812439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:07.813729 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:07.813736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:07.814027 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:07.814052 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 04:38:07.814118 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 04:38:07.814252 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":60,"thread_start_us":815,"threads_started":1}
I20260619 04:38:07.814709 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:07.793134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59310 (local address 127.0.97.3:44203)
0619 04:38:07.807628 (+ 14494us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:07.807635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:07.807664 (+ 29us) server_negotiation.cc:407] Connection header received
0619 04:38:07.807746 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:07.807752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:07.807944 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:07.808115 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:07.808765 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:07.810359 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:07.812620 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:07.813545 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:07.813665 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:07.813946 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:07.814195 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 04:38:07.814345 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14350,"thread_start_us":15192,"threads_started":1}
I20260619 04:38:07.816934 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:07.794510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45788 (local address 127.0.97.2:37939)
0619 04:38:07.809201 (+ 14691us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:07.809210 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:07.809243 (+ 33us) server_negotiation.cc:407] Connection header received
0619 04:38:07.809337 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:07.809345 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:07.809577 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:07.809760 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:07.810381 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:07.812043 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:07.814246 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:07.815569 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:07.815710 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:07.816322 (+ 612us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:07.816395 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 04:38:07.816580 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14515,"thread_start_us":14370,"threads_started":1}
I20260619 04:38:07.817374 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:07.794136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37939 (local address 127.0.0.1:45788)
0619 04:38:07.805387 (+ 11251us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:07.805406 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:07.805505 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:07.810019 (+ 4514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:07.810025 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:07.810049 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:07.810227 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:07.810237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:07.812211 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:07.812216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:07.814076 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:07.814090 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:07.815748 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:07.815754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:07.815929 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:07.815962 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 04:38:07.816040 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 04:38:07.816161 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11140,"mutex_wait_us":63,"thread_start_us":343,"threads_started":1}
I20260619 04:38:07.818512 19563 tablet_service.cc:1467] Processing CreateTablet for tablet 982e51c3f9084e519b85ccce4bb38644 (DEFAULT_TABLE table=test-table [id=f4d76baf491146a78bd3ec4786574c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:38:07.819757 19563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 982e51c3f9084e519b85ccce4bb38644. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:07.824226 19433 tablet_service.cc:1467] Processing CreateTablet for tablet 982e51c3f9084e519b85ccce4bb38644 (DEFAULT_TABLE table=test-table [id=f4d76baf491146a78bd3ec4786574c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:38:07.825649 19433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 982e51c3f9084e519b85ccce4bb38644. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:07.826618 19646 tablet_bootstrap.cc:492] T 982e51c3f9084e519b85ccce4bb38644 P 33559e87a0404e0c8ea4c695dfe07427: Bootstrap starting.
I20260619 04:38:07.830154 19646 tablet_bootstrap.cc:654] T 982e51c3f9084e519b85ccce4bb38644 P 33559e87a0404e0c8ea4c695dfe07427: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:07.831344 19646 log.cc:824] T 982e51c3f9084e519b85ccce4bb38644 P 33559e87a0404e0c8ea4c695dfe07427: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:07.833087 19648 tablet_bootstrap.cc:492] T 982e51c3f9084e519b85ccce4bb38644 P 6e59595e00e640a8b8ee64c509ef8447: Bootstrap starting.
I20260619 04:38:07.835208 19646 tablet_bootstrap.cc:492] T 982e51c3f9084e519b85ccce4bb38644 P 33559e87a0404e0c8ea4c695dfe07427: No bootstrap required, opened a new log
I20260619 04:38:07.835439 19646 ts_tablet_manager.cc:1397] T 982e51c3f9084e519b85ccce4bb38644 P 33559e87a0404e0c8ea4c695dfe07427: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 04:38:07.836602 19649 tablet_bootstrap.cc:492] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb: Bootstrap starting.
I20260619 04:38:07.838765 19648 tablet_bootstrap.cc:654] T 982e51c3f9084e519b85ccce4bb38644 P 6e59595e00e640a8b8ee64c509ef8447: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:07.839790 19649 tablet_bootstrap.cc:654] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:07.839824 19648 log.cc:824] T 982e51c3f9084e519b85ccce4bb38644 P 6e59595e00e640a8b8ee64c509ef8447: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:07.840924 19649 log.cc:824] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:07.842499 19648 tablet_bootstrap.cc:492] T 982e51c3f9084e519b85ccce4bb38644 P 6e59595e00e640a8b8ee64c509ef8447: No bootstrap required, opened a new log
I20260619 04:38:07.842734 19648 ts_tablet_manager.cc:1397] T 982e51c3f9084e519b85ccce4bb38644 P 6e59595e00e640a8b8ee64c509ef8447: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 04:38:07.843744 19649 tablet_bootstrap.cc:492] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb: No bootstrap required, opened a new log
I20260619 04:38:07.844017 19649 ts_tablet_manager.cc:1397] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 04:38:07.845052 19646 raft_consensus.cc:348] T 982e51c3f9084e519b85ccce4bb38644 P 33559e87a0404e0c8ea4c695dfe07427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33559e87a0404e0c8ea4c695dfe07427" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39311 } } peers { permanent_uuid: "6e59595e00e640a8b8ee64c509ef8447" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44203 } } peers { permanent_uuid: "82896a0ad07a445ebef24c9425b797bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37939 } }
I20260619 04:38:07.845505 19646 raft_consensus.cc:374] T 982e51c3f9084e519b85ccce4bb38644 P 33559e87a0404e0c8ea4c695dfe07427 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:07.845602 19646 raft_consensus.cc:729] T 982e51c3f9084e519b85ccce4bb38644 P 33559e87a0404e0c8ea4c695dfe07427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33559e87a0404e0c8ea4c695dfe07427, State: Initialized, Role: FOLLOWER
I20260619 04:38:07.846201 19646 consensus_queue.cc:260] T 982e51c3f9084e519b85ccce4bb38644 P 33559e87a0404e0c8ea4c695dfe07427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33559e87a0404e0c8ea4c695dfe07427" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39311 } } peers { permanent_uuid: "6e59595e00e640a8b8ee64c509ef8447" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44203 } } peers { permanent_uuid: "82896a0ad07a445ebef24c9425b797bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37939 } }
I20260619 04:38:07.847316 19367 heartbeater.cc:503] Master 127.0.97.62:35135 was elected leader, sending a full tablet report...
I20260619 04:38:07.847923 19646 ts_tablet_manager.cc:1428] T 982e51c3f9084e519b85ccce4bb38644 P 33559e87a0404e0c8ea4c695dfe07427: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260619 04:38:07.850679 19648 raft_consensus.cc:348] T 982e51c3f9084e519b85ccce4bb38644 P 6e59595e00e640a8b8ee64c509ef8447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33559e87a0404e0c8ea4c695dfe07427" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39311 } } peers { permanent_uuid: "6e59595e00e640a8b8ee64c509ef8447" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44203 } } peers { permanent_uuid: "82896a0ad07a445ebef24c9425b797bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37939 } }
I20260619 04:38:07.851011 19648 raft_consensus.cc:374] T 982e51c3f9084e519b85ccce4bb38644 P 6e59595e00e640a8b8ee64c509ef8447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:07.851121 19648 raft_consensus.cc:729] T 982e51c3f9084e519b85ccce4bb38644 P 6e59595e00e640a8b8ee64c509ef8447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e59595e00e640a8b8ee64c509ef8447, State: Initialized, Role: FOLLOWER
I20260619 04:38:07.851755 19648 consensus_queue.cc:260] T 982e51c3f9084e519b85ccce4bb38644 P 6e59595e00e640a8b8ee64c509ef8447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33559e87a0404e0c8ea4c695dfe07427" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39311 } } peers { permanent_uuid: "6e59595e00e640a8b8ee64c509ef8447" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44203 } } peers { permanent_uuid: "82896a0ad07a445ebef24c9425b797bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37939 } }
I20260619 04:38:07.852957 19628 heartbeater.cc:503] Master 127.0.97.62:35135 was elected leader, sending a full tablet report...
I20260619 04:38:07.853675 19649 raft_consensus.cc:348] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33559e87a0404e0c8ea4c695dfe07427" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39311 } } peers { permanent_uuid: "6e59595e00e640a8b8ee64c509ef8447" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44203 } } peers { permanent_uuid: "82896a0ad07a445ebef24c9425b797bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37939 } }
I20260619 04:38:07.854094 19649 raft_consensus.cc:374] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:07.854207 19649 raft_consensus.cc:729] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82896a0ad07a445ebef24c9425b797bb, State: Initialized, Role: FOLLOWER
I20260619 04:38:07.854765 19648 ts_tablet_manager.cc:1428] T 982e51c3f9084e519b85ccce4bb38644 P 6e59595e00e640a8b8ee64c509ef8447: Time spent starting tablet: real 0.012s user 0.004s sys 0.006s
I20260619 04:38:07.854730 19649 consensus_queue.cc:260] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33559e87a0404e0c8ea4c695dfe07427" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39311 } } peers { permanent_uuid: "6e59595e00e640a8b8ee64c509ef8447" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44203 } } peers { permanent_uuid: "82896a0ad07a445ebef24c9425b797bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37939 } }
I20260619 04:38:07.855890 19498 heartbeater.cc:503] Master 127.0.97.62:35135 was elected leader, sending a full tablet report...
I20260619 04:38:07.857228 19649 ts_tablet_manager.cc:1428] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260619 04:38:07.867237 19368 tablet.cc:2369] T 982e51c3f9084e519b85ccce4bb38644 P 33559e87a0404e0c8ea4c695dfe07427: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:38:07.989753 19629 tablet.cc:2369] T 982e51c3f9084e519b85ccce4bb38644 P 6e59595e00e640a8b8ee64c509ef8447: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:38:08.000339 19654 raft_consensus.cc:482] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:38:08.000680 19654 raft_consensus.cc:504] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33559e87a0404e0c8ea4c695dfe07427" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39311 } } peers { permanent_uuid: "6e59595e00e640a8b8ee64c509ef8447" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44203 } } peers { permanent_uuid: "82896a0ad07a445ebef24c9425b797bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37939 } }
I20260619 04:38:08.002370 19654 leader_election.cc:288] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33559e87a0404e0c8ea4c695dfe07427 (127.0.97.1:39311), 6e59595e00e640a8b8ee64c509ef8447 (127.0.97.3:44203)
I20260619 04:38:08.008796 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:08.002757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33271 (local address 127.0.97.3:44203)
0619 04:38:08.003053 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:08.003060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:08.003671 (+ 611us) server_negotiation.cc:407] Connection header received
0619 04:38:08.003821 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:08.003827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:08.003903 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:08.003990 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:08.004665 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:08.005385 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:08.007101 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:08.007867 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:08.007993 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:08.008085 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:08.008420 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 04:38:08.008568 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260619 04:38:08.008803 19641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:08.002922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44245 (local address 127.0.97.1:39311)
0619 04:38:08.003140 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:08.003146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:08.003423 (+ 277us) server_negotiation.cc:407] Connection header received
0619 04:38:08.003633 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:08.003639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:08.003725 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:08.003810 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:08.004489 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:08.005212 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:08.007080 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:08.007815 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:08.007906 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:08.007988 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:08.008424 (+ 436us) server_negotiation.cc:299] Negotiation successful
0619 04:38:08.008565 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 04:38:08.008899 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:08.002580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39311 (local address 127.0.97.2:44245)
0619 04:38:08.003300 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:08.003322 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:08.003484 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:08.003830 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:08.003835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:08.003868 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:08.004326 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:08.004340 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:08.005338 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:08.005344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:08.006896 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:08.006908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:08.008010 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:08.008017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:08.008255 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:08.008306 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 04:38:08.008383 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 04:38:08.008523 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":124,"thread_start_us":360,"threads_started":1}
I20260619 04:38:08.008927 19656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:08.002713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44203 (local address 127.0.97.2:33271)
0619 04:38:08.003564 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:08.003581 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:08.003704 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:08.004027 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:08.004030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:08.004054 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:08.004508 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:08.004520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:08.005523 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:08.005527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:08.006920 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:08.006931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:08.008051 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:08.008056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:08.008252 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:08.008297 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 04:38:08.008370 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 04:38:08.008523 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":60,"thread_start_us":544,"threads_started":1}
I20260619 04:38:08.009883 19322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "982e51c3f9084e519b85ccce4bb38644" candidate_uuid: "82896a0ad07a445ebef24c9425b797bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33559e87a0404e0c8ea4c695dfe07427" is_pre_election: true
I20260619 04:38:08.009956 19583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "982e51c3f9084e519b85ccce4bb38644" candidate_uuid: "82896a0ad07a445ebef24c9425b797bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e59595e00e640a8b8ee64c509ef8447" is_pre_election: true
I20260619 04:38:08.010362 19322 raft_consensus.cc:2454] T 982e51c3f9084e519b85ccce4bb38644 P 33559e87a0404e0c8ea4c695dfe07427 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82896a0ad07a445ebef24c9425b797bb in term 0.
I20260619 04:38:08.010372 19583 raft_consensus.cc:2454] T 982e51c3f9084e519b85ccce4bb38644 P 6e59595e00e640a8b8ee64c509ef8447 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82896a0ad07a445ebef24c9425b797bb in term 0.
I20260619 04:38:08.010921 19389 leader_election.cc:302] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e59595e00e640a8b8ee64c509ef8447, 82896a0ad07a445ebef24c9425b797bb; no voters:
I20260619 04:38:08.011278 19654 raft_consensus.cc:2781] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:38:08.011399 19654 raft_consensus.cc:482] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:38:08.011520 19654 raft_consensus.cc:3037] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:08.013765 19654 raft_consensus.cc:504] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33559e87a0404e0c8ea4c695dfe07427" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39311 } } peers { permanent_uuid: "6e59595e00e640a8b8ee64c509ef8447" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44203 } } peers { permanent_uuid: "82896a0ad07a445ebef24c9425b797bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37939 } }
I20260619 04:38:08.014365 19654 leader_election.cc:288] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb [CANDIDATE]: Term 1 election: Requested vote from peers 33559e87a0404e0c8ea4c695dfe07427 (127.0.97.1:39311), 6e59595e00e640a8b8ee64c509ef8447 (127.0.97.3:44203)
I20260619 04:38:08.014786 19322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "982e51c3f9084e519b85ccce4bb38644" candidate_uuid: "82896a0ad07a445ebef24c9425b797bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33559e87a0404e0c8ea4c695dfe07427"
I20260619 04:38:08.014974 19322 raft_consensus.cc:3037] T 982e51c3f9084e519b85ccce4bb38644 P 33559e87a0404e0c8ea4c695dfe07427 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:08.014912 19583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "982e51c3f9084e519b85ccce4bb38644" candidate_uuid: "82896a0ad07a445ebef24c9425b797bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e59595e00e640a8b8ee64c509ef8447"
I20260619 04:38:08.015328 19583 raft_consensus.cc:3037] T 982e51c3f9084e519b85ccce4bb38644 P 6e59595e00e640a8b8ee64c509ef8447 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:08.017197 19322 raft_consensus.cc:2454] T 982e51c3f9084e519b85ccce4bb38644 P 33559e87a0404e0c8ea4c695dfe07427 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82896a0ad07a445ebef24c9425b797bb in term 1.
I20260619 04:38:08.017568 19387 leader_election.cc:302] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33559e87a0404e0c8ea4c695dfe07427, 82896a0ad07a445ebef24c9425b797bb; no voters:
I20260619 04:38:08.017664 19583 raft_consensus.cc:2454] T 982e51c3f9084e519b85ccce4bb38644 P 6e59595e00e640a8b8ee64c509ef8447 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82896a0ad07a445ebef24c9425b797bb in term 1.
I20260619 04:38:08.017827 19654 raft_consensus.cc:2781] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:08.018253 19654 raft_consensus.cc:686] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb [term 1 LEADER]: Becoming Leader. State: Replica: 82896a0ad07a445ebef24c9425b797bb, State: Running, Role: LEADER
I20260619 04:38:08.018695 19654 consensus_queue.cc:237] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33559e87a0404e0c8ea4c695dfe07427" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39311 } } peers { permanent_uuid: "6e59595e00e640a8b8ee64c509ef8447" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44203 } } peers { permanent_uuid: "82896a0ad07a445ebef24c9425b797bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37939 } }
I20260619 04:38:08.023285 19181 catalog_manager.cc:5495] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 82896a0ad07a445ebef24c9425b797bb (127.0.97.2). New cstate: current_term: 1 leader_uuid: "82896a0ad07a445ebef24c9425b797bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33559e87a0404e0c8ea4c695dfe07427" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e59595e00e640a8b8ee64c509ef8447" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82896a0ad07a445ebef24c9425b797bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37939 } health_report { overall_health: HEALTHY } } }
I20260619 04:38:08.042264 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:08.037338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45804 (local address 127.0.97.2:37939)
0619 04:38:08.037536 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:08.037541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:08.037613 (+ 72us) server_negotiation.cc:407] Connection header received
0619 04:38:08.037772 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:08.037775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:08.037855 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:08.037970 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:08.038280 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:08.038948 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:08.040315 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:08.040750 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:08.040835 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:08.041303 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:08.041800 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:08.041920 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:08.042048 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 04:38:08.042082 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 04:38:08.050091 19583 raft_consensus.cc:1261] T 982e51c3f9084e519b85ccce4bb38644 P 6e59595e00e640a8b8ee64c509ef8447 [term 1 FOLLOWER]: Refusing update from remote peer 82896a0ad07a445ebef24c9425b797bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:38:08.050124 19322 raft_consensus.cc:1261] T 982e51c3f9084e519b85ccce4bb38644 P 33559e87a0404e0c8ea4c695dfe07427 [term 1 FOLLOWER]: Refusing update from remote peer 82896a0ad07a445ebef24c9425b797bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:38:08.051015 19654 consensus_queue.cc:1035] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "33559e87a0404e0c8ea4c695dfe07427" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:38:08.051343 19657 consensus_queue.cc:1035] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e59595e00e640a8b8ee64c509ef8447" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 04:38:08.060863 19499 tablet.cc:2369] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:38:08.065368 19662 mvcc.cc:205] Tried to move back new op lower bound from 7298432565444079616 to 7298432565328273408. Current Snapshot: MvccSnapshot[applied={T|T < 7298432565444079616}]
I20260619 04:38:08.076807 19181 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:38:08.077024 19181 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:38:08.080631 19181 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dbc3f4cf6e9249d6b39538a0fb9af190: Sending DeleteTablet for 3 replicas of tablet 982e51c3f9084e519b85ccce4bb38644
I20260619 04:38:08.081936 19302 tablet_service.cc:1514] Processing DeleteTablet for tablet 982e51c3f9084e519b85ccce4bb38644 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:38:08 UTC) from {username='slave'} at 127.0.0.1:49634
I20260619 04:38:08.081988 19563 tablet_service.cc:1514] Processing DeleteTablet for tablet 982e51c3f9084e519b85ccce4bb38644 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:38:08 UTC) from {username='slave'} at 127.0.0.1:59310
I20260619 04:38:08.082369 19433 tablet_service.cc:1514] Processing DeleteTablet for tablet 982e51c3f9084e519b85ccce4bb38644 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:38:08 UTC) from {username='slave'} at 127.0.0.1:45788
I20260619 04:38:08.083853 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 19240
I20260619 04:38:08.094564 19671 tablet_replica.cc:330] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb: stopping tablet replica
I20260619 04:38:08.095228 19671 raft_consensus.cc:2229] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:38:08.096017 19671 raft_consensus.cc:2258] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:38:08.096663 19670 tablet_replica.cc:330] T 982e51c3f9084e519b85ccce4bb38644 P 6e59595e00e640a8b8ee64c509ef8447: stopping tablet replica
I20260619 04:38:08.097460 19670 raft_consensus.cc:2229] T 982e51c3f9084e519b85ccce4bb38644 P 6e59595e00e640a8b8ee64c509ef8447 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:38:08.099843 19671 ts_tablet_manager.cc:1905] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:38:08.100430 19670 raft_consensus.cc:2258] T 982e51c3f9084e519b85ccce4bb38644 P 6e59595e00e640a8b8ee64c509ef8447 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:38:08.105244 19670 ts_tablet_manager.cc:1905] T 982e51c3f9084e519b85ccce4bb38644 P 6e59595e00e640a8b8ee64c509ef8447: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:38:08.107120 19669 tablet_replica.cc:330] T 982e51c3f9084e519b85ccce4bb38644 P 33559e87a0404e0c8ea4c695dfe07427: stopping tablet replica
I20260619 04:38:08.107532 19669 raft_consensus.cc:2229] T 982e51c3f9084e519b85ccce4bb38644 P 33559e87a0404e0c8ea4c695dfe07427 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:38:08.108023 19669 raft_consensus.cc:2258] T 982e51c3f9084e519b85ccce4bb38644 P 33559e87a0404e0c8ea4c695dfe07427 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:38:08.108777 19641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:08.085391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49648 (local address 127.0.97.1:39311)
0619 04:38:08.085583 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:08.085589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:08.100129 (+ 14540us) server_negotiation.cc:407] Connection header received
0619 04:38:08.100683 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:08.100689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:08.100852 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:08.100937 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:08.101642 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:08.102818 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:08.104808 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:08.105450 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:08.105545 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:08.106986 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:08.106999 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:08.107017 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:08.107805 (+ 788us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:08.107868 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:08.107904 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:08.108024 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:08.108239 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 04:38:08.108354 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 04:38:08.109925 19362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:08.112913 19670 ts_tablet_manager.cc:1918] T 982e51c3f9084e519b85ccce4bb38644 P 6e59595e00e640a8b8ee64c509ef8447: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:38:08.113126 19670 log.cc:1196] T 982e51c3f9084e519b85ccce4bb38644 P 6e59595e00e640a8b8ee64c509ef8447: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/982e51c3f9084e519b85ccce4bb38644
I20260619 04:38:08.113713 19670 ts_tablet_manager.cc:1939] T 982e51c3f9084e519b85ccce4bb38644 P 6e59595e00e640a8b8ee64c509ef8447: Deleting consensus metadata
I20260619 04:38:08.115651 19168 catalog_manager.cc:4841] TS 6e59595e00e640a8b8ee64c509ef8447 (127.0.97.3:44203): tablet 982e51c3f9084e519b85ccce4bb38644 (table test-table [id=f4d76baf491146a78bd3ec4786574c70]) successfully deleted
I20260619 04:38:08.115744 19671 ts_tablet_manager.cc:1918] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:38:08.115906 19671 log.cc:1196] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/982e51c3f9084e519b85ccce4bb38644
I20260619 04:38:08.116338 19671 ts_tablet_manager.cc:1939] T 982e51c3f9084e519b85ccce4bb38644 P 82896a0ad07a445ebef24c9425b797bb: Deleting consensus metadata
I20260619 04:38:08.117772 19168 catalog_manager.cc:4841] TS 82896a0ad07a445ebef24c9425b797bb (127.0.97.2:37939): tablet 982e51c3f9084e519b85ccce4bb38644 (table test-table [id=f4d76baf491146a78bd3ec4786574c70]) successfully deleted
I20260619 04:38:08.110425 19669 ts_tablet_manager.cc:1905] T 982e51c3f9084e519b85ccce4bb38644 P 33559e87a0404e0c8ea4c695dfe07427: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:38:08.205564 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 19240
W20260619 04:38:08.226907 19166 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39311 (error 108)
I20260619 04:38:08.227048 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 19371
I20260619 04:38:08.228902 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:08.228344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39311 (local address 127.0.0.1:49664)
0619 04:38:08.228626 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:08.228749 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 04:38:08.229143 19166 catalog_manager.cc:4584] TS 33559e87a0404e0c8ea4c695dfe07427 (127.0.97.1:39311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 982e51c3f9084e519b85ccce4bb38644: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39311: connect: Connection refused (error 111)
I20260619 04:38:08.235488 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:08.228598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45806 (local address 127.0.97.2:37939)
0619 04:38:08.228793 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:08.228799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:08.228819 (+ 20us) server_negotiation.cc:407] Connection header received
0619 04:38:08.228892 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:08.228897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:08.229003 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:08.229101 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:08.229755 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:08.230520 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:08.232306 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:08.232814 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:08.232917 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:08.234049 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:08.234057 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:08.234071 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:08.234688 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:08.234752 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:08.234779 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:08.234860 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:08.235054 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 04:38:08.235141 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 04:38:08.236245 19493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:08.282683 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:08.282158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39311 (local address 127.0.0.1:49666)
0619 04:38:08.282393 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:08.282502 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 04:38:08.283699 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:08.283309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39311 (local address 127.0.0.1:49674)
0619 04:38:08.283447 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:08.283529 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 04:38:08.344280 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:08.343774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39311 (local address 127.0.0.1:49680)
0619 04:38:08.344002 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:08.344102 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 04:38:08.345376 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:08.345000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39311 (local address 127.0.0.1:49694)
0619 04:38:08.345157 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:08.345247 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 04:38:08.346822 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 19371
I20260619 04:38:08.361660 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 19501
I20260619 04:38:08.367981 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:08.362370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.0.97.3:44203)
0619 04:38:08.362543 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:08.362547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:08.362582 (+ 35us) server_negotiation.cc:407] Connection header received
0619 04:38:08.362746 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:08.362750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:08.362838 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:08.362918 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:08.363250 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:08.363946 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:08.365304 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:08.365687 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:08.365775 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:08.366777 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:08.366786 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:08.366801 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:08.367347 (+ 546us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:08.367396 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:08.367415 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:08.367497 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:08.367584 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 04:38:08.367680 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 04:38:08.368633 19623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:08.426170 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:08.425634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39311 (local address 127.0.0.1:49708)
0619 04:38:08.425903 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:08.426009 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 04:38:08.427237 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:08.426822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39311 (local address 127.0.0.1:49710)
0619 04:38:08.427021 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:08.427102 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260619 04:38:08.454677 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 19501
I20260619 04:38:08.467649 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 19150
I20260619 04:38:08.468772 19211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:08.574894 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 19150
2026-06-19T04:38:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2718 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:38:08 dist-test-slave-n3tq krb5kdc[19680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:38:08 dist-test-slave-n3tq krb5kdc[19680](info): set up 2 sockets
Jun 19 04:38:08 dist-test-slave-n3tq krb5kdc[19680](info): commencing operation
krb5kdc: starting...
W20260619 04:38:09.584394 388 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:38:09 dist-test-slave-n3tq krb5kdc[19680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:38:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:38:09.817445 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35551
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:10.134310 19693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:10.134585 19693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:10.134716 19693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:10.141670 19693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:10.141764 19693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:10.141836 19693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:10.141911 19693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:10.141969 19693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:10.151331 19693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:10.152667 19693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:10.153901 19693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:10.164939 19701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:10.164963 19699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:10.165215 19698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:10.165685 19693 server_base.cc:1034] running on GCE node
I20260619 04:38:10.166438 19693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:10.167804 19693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:10.169001 19693 hybrid_clock.cc:648] HybridClock initialized: now 1781843890168958 us; error 28 us; skew 500 ppm
I20260619 04:38:10.169317 19693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:10 dist-test-slave-n3tq krb5kdc[19680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:10.178768 19693 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:38:10.180279 19693 webserver.cc:460] Webserver started at http://127.0.97.62:34773/ using document root <none> and password file <none>
I20260619 04:38:10.180995 19693 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:10.181109 19693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:10.181406 19693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:10.184290 19693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "95553cfdb32046e09e30948e539aa464"
format_stamp: "Formatted at 2026-06-19 04:38:10 on dist-test-slave-n3tq"
server_key: "31289106e2a95161bb4381c11cecfdca"
server_key_iv: "4a507388b50cec676bd5d10d95d66ffd"
server_key_version: "encryptionkey@0"
I20260619 04:38:10.185040 19693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "95553cfdb32046e09e30948e539aa464"
format_stamp: "Formatted at 2026-06-19 04:38:10 on dist-test-slave-n3tq"
server_key: "31289106e2a95161bb4381c11cecfdca"
server_key_iv: "4a507388b50cec676bd5d10d95d66ffd"
server_key_version: "encryptionkey@0"
I20260619 04:38:10.190156 19693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 04:38:10.193599 19708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:10.194963 19693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:38:10.195137 19693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "95553cfdb32046e09e30948e539aa464"
format_stamp: "Formatted at 2026-06-19 04:38:10 on dist-test-slave-n3tq"
server_key: "31289106e2a95161bb4381c11cecfdca"
server_key_iv: "4a507388b50cec676bd5d10d95d66ffd"
server_key_version: "encryptionkey@0"
I20260619 04:38:10.195325 19693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:10.223412 19693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:10.227828 19693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:10.228106 19693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:10.245378 19693 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35551
I20260619 04:38:10.245427 19759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35551 every 8 connection(s)
I20260619 04:38:10.246722 19693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:10.248966 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 19693
I20260619 04:38:10.249110 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:10.249532 388 external_mini_cluster.cc:1428] Setting key 1b02bb2cc8837b4b9169abeb36c6d7e0
I20260619 04:38:10.250730 19760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 04:38:10 dist-test-slave-n3tq krb5kdc[19680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:38:10.263796 19760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464: Bootstrap starting.
I20260619 04:38:10.267603 19760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:10.268824 19760 log.cc:824] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:10.272089 19760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464: No bootstrap required, opened a new log
I20260619 04:38:10.272506 19763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:10.251285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.0.97.62:35551)
0619 04:38:10.252065 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:10.252080 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:10.252138 (+ 58us) server_negotiation.cc:407] Connection header received
0619 04:38:10.254031 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:10.254060 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:10.254584 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:10.255028 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:10.256271 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:10.258317 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:10.259617 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:10.260605 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:10.260806 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:10.264618 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:10.264629 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:10.264648 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:10.264695 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:10.268306 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:10.268987 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:10.268996 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:10.269003 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:10.269198 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:10.269598 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:10.269601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:10.269603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:10.270122 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:10.270421 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:10.270781 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 04:38:10.271196 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":312,"threads_started":1}
I20260619 04:38:10.279990 19760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95553cfdb32046e09e30948e539aa464" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35551 } }
I20260619 04:38:10.280341 19760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:10.280413 19760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95553cfdb32046e09e30948e539aa464, State: Initialized, Role: FOLLOWER
I20260619 04:38:10.281018 19760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95553cfdb32046e09e30948e539aa464" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35551 } }
I20260619 04:38:10.281235 19760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:38:10.281330 19760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:38:10.281456 19760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:10.283614 19760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95553cfdb32046e09e30948e539aa464" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35551 } }
I20260619 04:38:10.284101 19760 leader_election.cc:302] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95553cfdb32046e09e30948e539aa464; no voters:
I20260619 04:38:10.284678 19760 leader_election.cc:288] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:38:10.284783 19765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:10.285405 19765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464 [term 1 LEADER]: Becoming Leader. State: Replica: 95553cfdb32046e09e30948e539aa464, State: Running, Role: LEADER
I20260619 04:38:10.286043 19765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95553cfdb32046e09e30948e539aa464" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35551 } }
I20260619 04:38:10.286811 19760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:10.288861 19766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95553cfdb32046e09e30948e539aa464" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95553cfdb32046e09e30948e539aa464" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35551 } } }
I20260619 04:38:10.289054 19767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95553cfdb32046e09e30948e539aa464. Latest consensus state: current_term: 1 leader_uuid: "95553cfdb32046e09e30948e539aa464" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95553cfdb32046e09e30948e539aa464" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35551 } } }
I20260619 04:38:10.289649 19767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:10.289536 19766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:10.292943 19772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:10.298108 19772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:10.307665 19772 catalog_manager.cc:1347] Generated new cluster ID: 965da7f1741d4e8287d3300b76955dee
I20260619 04:38:10.307790 19772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:10.320631 19772 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:10.321931 19772 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:10.336238 19772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464: Generated new TSK 0
I20260619 04:38:10.337441 19772 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 04:38:10.405292 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35551
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:10.724674 19786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:10.724946 19786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:10.725088 19786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:10.732188 19786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:10.732288 19786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:10.732401 19786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:38:10.742139 19786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:10.743292 19786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:10.744525 19786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:10.756860 19791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:10.756886 19792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:10.756964 19794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:10.757561 19786 server_base.cc:1034] running on GCE node
I20260619 04:38:10.758078 19786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:10.758920 19786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:10.760115 19786 hybrid_clock.cc:648] HybridClock initialized: now 1781843890760066 us; error 40 us; skew 500 ppm
I20260619 04:38:10.760402 19786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:10 dist-test-slave-n3tq krb5kdc[19680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:10.763273 19786 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 04:38:10.764658 19786 webserver.cc:460] Webserver started at http://127.0.97.1:34607/ using document root <none> and password file <none>
I20260619 04:38:10.765308 19786 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:10.765420 19786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:10.765697 19786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:10.768431 19786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "b4cb80e9813e44b0bebbfba96385942a"
format_stamp: "Formatted at 2026-06-19 04:38:10 on dist-test-slave-n3tq"
server_key: "223d2b21d1842995149f72f208195647"
server_key_iv: "04ab46057ab9fce0ebc26f7b84097379"
server_key_version: "encryptionkey@0"
I20260619 04:38:10.769210 19786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b4cb80e9813e44b0bebbfba96385942a"
format_stamp: "Formatted at 2026-06-19 04:38:10 on dist-test-slave-n3tq"
server_key: "223d2b21d1842995149f72f208195647"
server_key_iv: "04ab46057ab9fce0ebc26f7b84097379"
server_key_version: "encryptionkey@0"
I20260619 04:38:10.774098 19786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 04:38:10.777308 19801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:10.778554 19786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:38:10.778700 19786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "b4cb80e9813e44b0bebbfba96385942a"
format_stamp: "Formatted at 2026-06-19 04:38:10 on dist-test-slave-n3tq"
server_key: "223d2b21d1842995149f72f208195647"
server_key_iv: "04ab46057ab9fce0ebc26f7b84097379"
server_key_version: "encryptionkey@0"
I20260619 04:38:10.778868 19786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:10.796315 19786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:10.800431 19786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:10.800729 19786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:10.801657 19786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:38:10.802947 19786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:38:10.803025 19786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:10.803143 19786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:38:10.803194 19786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:10.832602 19786 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32997
I20260619 04:38:10.832619 19913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32997 every 8 connection(s)
I20260619 04:38:10.833806 19786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:38:10.837005 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 19786
I20260619 04:38:10.837181 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:38:10.837736 388 external_mini_cluster.cc:1428] Setting key 0817010bfbae03bf3eb558d822337c6d
Jun 19 04:38:10 dist-test-slave-n3tq krb5kdc[19680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:38:10.854359 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:10.836212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54969 (local address 127.0.97.62:35551)
0619 04:38:10.836671 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:10.836677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:10.837739 (+ 1062us) server_negotiation.cc:407] Connection header received
0619 04:38:10.839994 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:10.839998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:10.840063 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:10.840146 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:10.842079 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:10.842769 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:10.844741 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:10.845128 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:10.845218 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:10.849099 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:10.849104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:10.849107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:10.849139 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:10.851361 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:10.852027 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:10.852031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:10.852033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:10.852146 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:10.852582 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:10.852586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:10.852588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:10.852920 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:10.853101 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:10.853775 (+ 674us) server_negotiation.cc:299] Negotiation successful
0619 04:38:10.854007 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":222,"threads_started":1}
I20260619 04:38:10.855274 19917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:10.836770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35551 (local address 127.0.97.1:54969)
0619 04:38:10.837561 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:10.837614 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:10.839697 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:10.840434 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:10.840445 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:10.841138 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:10.841898 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:10.841920 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:10.842921 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:10.842926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:10.844625 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:10.844639 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:10.845333 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:10.845339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:10.845614 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:10.846300 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:10.846326 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:10.848869 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:10.851489 (+ 2620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:10.851498 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:10.851515 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:10.851903 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:10.852242 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:10.852247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:10.852250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:10.852439 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:10.853063 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:10.853071 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:10.853414 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 04:38:10.853651 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 04:38:10.854156 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":291,"threads_started":1}
I20260619 04:38:10.856793 19914 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35551
I20260619 04:38:10.857074 19914 heartbeater.cc:464] Registering TS with master...
I20260619 04:38:10.857811 19914 heartbeater.cc:511] Master 127.0.97.62:35551 requested a full tablet report, sending...
I20260619 04:38:10.860064 19725 ts_manager.cc:194] Registered new tserver with Master: b4cb80e9813e44b0bebbfba96385942a (127.0.97.1:32997)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 04:38:10.862067 19725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 04:38:10.888779 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35551
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:11.200702 19920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:11.200999 19920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:11.201136 19920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:11.208209 19920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:11.208328 19920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:11.208499 19920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:38:11.218114 19920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:11.219450 19920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:11.220737 19920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:11.232874 19925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:11.232897 19926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:11.233107 19928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:11.233862 19920 server_base.cc:1034] running on GCE node
I20260619 04:38:11.234400 19920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:11.235216 19920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:11.236429 19920 hybrid_clock.cc:648] HybridClock initialized: now 1781843891236380 us; error 38 us; skew 500 ppm
I20260619 04:38:11.236780 19920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:11 dist-test-slave-n3tq krb5kdc[19680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:11.239907 19920 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 04:38:11.241392 19920 webserver.cc:460] Webserver started at http://127.0.97.2:39715/ using document root <none> and password file <none>
I20260619 04:38:11.242061 19920 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:11.242173 19920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:11.242471 19920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:11.245687 19920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "8b56f0cb92d54f4a80333677b86ede02"
format_stamp: "Formatted at 2026-06-19 04:38:11 on dist-test-slave-n3tq"
server_key: "628bcf00ef926e92651c9c2472743696"
server_key_iv: "16a0048891b139b183a4717eee3cd662"
server_key_version: "encryptionkey@0"
I20260619 04:38:11.246654 19920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8b56f0cb92d54f4a80333677b86ede02"
format_stamp: "Formatted at 2026-06-19 04:38:11 on dist-test-slave-n3tq"
server_key: "628bcf00ef926e92651c9c2472743696"
server_key_iv: "16a0048891b139b183a4717eee3cd662"
server_key_version: "encryptionkey@0"
I20260619 04:38:11.253082 19920 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 04:38:11.257340 19935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:11.259069 19920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260619 04:38:11.259240 19920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "8b56f0cb92d54f4a80333677b86ede02"
format_stamp: "Formatted at 2026-06-19 04:38:11 on dist-test-slave-n3tq"
server_key: "628bcf00ef926e92651c9c2472743696"
server_key_iv: "16a0048891b139b183a4717eee3cd662"
server_key_version: "encryptionkey@0"
I20260619 04:38:11.259558 19920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:11.281426 19920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:11.285619 19920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:11.285848 19920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:11.286809 19920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:38:11.288278 19920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:38:11.288370 19920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:11.288478 19920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:38:11.288532 19920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 04:38:11.318642 19920 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34997
I20260619 04:38:11.318691 20047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34997 every 8 connection(s)
I20260619 04:38:11.319875 19920 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:38:11.320899 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 19920
I20260619 04:38:11.321074 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:38:11.321642 388 external_mini_cluster.cc:1428] Setting key 48a1e52ac5b844b84f36b60e585e1cbc
Jun 19 04:38:11 dist-test-slave-n3tq krb5kdc[19680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:38:11.339891 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:11.322658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34107 (local address 127.0.97.62:35551)
0619 04:38:11.322823 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:11.322827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:11.324127 (+ 1300us) server_negotiation.cc:407] Connection header received
0619 04:38:11.325971 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:11.325976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:11.326062 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:11.326146 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:11.327655 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:11.328385 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:11.329944 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:11.330375 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:11.330497 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:11.334310 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:11.334315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:11.334318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:11.334357 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:11.336954 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:11.337667 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:11.337671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:11.337673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:11.337813 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:11.338213 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:11.338216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:11.338218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:11.338507 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:11.338635 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:11.339310 (+ 675us) server_negotiation.cc:299] Negotiation successful
0619 04:38:11.339533 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 04:38:11.340907 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:11.323163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35551 (local address 127.0.97.2:34107)
0619 04:38:11.323976 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:11.324019 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:11.325706 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:11.326390 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:11.326398 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:11.326923 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:11.327491 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:11.327508 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:11.328553 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:11.328557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:11.329796 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:11.329808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:11.330609 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:11.330614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:11.330841 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:11.331576 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:11.331609 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:11.334087 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:11.337084 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:11.337093 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:11.337110 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:11.337542 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:11.337916 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:11.337922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:11.337925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:11.338102 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:11.338628 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:11.338636 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:11.338960 (+ 324us) client_negotiation.cc:769] Sending connection context
0619 04:38:11.339190 (+ 230us) client_negotiation.cc:240] Negotiation successful
0619 04:38:11.339717 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":334,"threads_started":1}
I20260619 04:38:11.342360 20048 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35551
I20260619 04:38:11.342676 20048 heartbeater.cc:464] Registering TS with master...
I20260619 04:38:11.343362 20048 heartbeater.cc:511] Master 127.0.97.62:35551 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 04:38:11.344825 19725 ts_manager.cc:194] Registered new tserver with Master: 8b56f0cb92d54f4a80333677b86ede02 (127.0.97.2:34997)
I20260619 04:38:11.345695 19725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 04:38:11.371315 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35551
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:11.682122 20053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:11.682417 20053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:11.682511 20053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:11.689775 20053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:11.689879 20053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:11.689996 20053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:38:11.699682 20053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:11.701027 20053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:11.702257 20053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:11.715116 20058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:11.715272 20059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:11.715113 20061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:11.715941 20053 server_base.cc:1034] running on GCE node
I20260619 04:38:11.716598 20053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:11.717538 20053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:11.718770 20053 hybrid_clock.cc:648] HybridClock initialized: now 1781843891718717 us; error 43 us; skew 500 ppm
I20260619 04:38:11.719094 20053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:11 dist-test-slave-n3tq krb5kdc[19680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:11.722188 20053 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 04:38:11.723766 20053 webserver.cc:460] Webserver started at http://127.0.97.3:35737/ using document root <none> and password file <none>
I20260619 04:38:11.724479 20053 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:11.724599 20053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:11.724905 20053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:11.727813 20053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "283823fc897943aabb15cbf0b012e0fe"
format_stamp: "Formatted at 2026-06-19 04:38:11 on dist-test-slave-n3tq"
server_key: "2a4004bee9b11df8038c2038176e65c8"
server_key_iv: "8ab86d5a2aeb30575dcb4d45469c33f6"
server_key_version: "encryptionkey@0"
I20260619 04:38:11.728615 20053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "283823fc897943aabb15cbf0b012e0fe"
format_stamp: "Formatted at 2026-06-19 04:38:11 on dist-test-slave-n3tq"
server_key: "2a4004bee9b11df8038c2038176e65c8"
server_key_iv: "8ab86d5a2aeb30575dcb4d45469c33f6"
server_key_version: "encryptionkey@0"
I20260619 04:38:11.733534 20053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 04:38:11.736771 20068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:11.737982 20053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:38:11.738117 20053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "283823fc897943aabb15cbf0b012e0fe"
format_stamp: "Formatted at 2026-06-19 04:38:11 on dist-test-slave-n3tq"
server_key: "2a4004bee9b11df8038c2038176e65c8"
server_key_iv: "8ab86d5a2aeb30575dcb4d45469c33f6"
server_key_version: "encryptionkey@0"
I20260619 04:38:11.738282 20053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:11.760074 20053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:11.764581 20053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:11.764873 20053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:11.765857 20053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:38:11.767161 20053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:38:11.767233 20053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:11.767323 20053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:38:11.767364 20053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:11.798085 20053 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35267
I20260619 04:38:11.798238 20180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35267 every 8 connection(s)
I20260619 04:38:11.799422 20053 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:38:11.803613 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 20053
I20260619 04:38:11.803750 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:38:11.804165 388 external_mini_cluster.cc:1428] Setting key 006a2e94c39b37d229a60a123d444fe2
Jun 19 04:38:11 dist-test-slave-n3tq krb5kdc[19680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:38:11.818585 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:11.801880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56811 (local address 127.0.97.62:35551)
0619 04:38:11.802064 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:11.802068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:11.803226 (+ 1158us) server_negotiation.cc:407] Connection header received
0619 04:38:11.805262 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:11.805268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:11.805398 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:11.805495 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:11.807311 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:11.808044 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:11.809519 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:11.809903 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:11.809993 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:11.813093 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:11.813098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:11.813101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:11.813132 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:11.815532 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:11.816232 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:11.816236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:11.816238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:11.816392 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:11.816788 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:11.816792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:11.816794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:11.817068 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:11.817197 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:11.818015 (+ 818us) server_negotiation.cc:299] Negotiation successful
0619 04:38:11.818229 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 04:38:11.819612 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:11.802337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35551 (local address 127.0.97.3:56811)
0619 04:38:11.803065 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:11.803113 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:11.804953 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:11.805702 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:11.805713 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:11.806404 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:11.807156 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:11.807175 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:11.808157 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:11.808161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:11.809412 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:11.809424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:11.810061 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:11.810065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:11.810247 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:11.810852 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:11.810883 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:11.812889 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:11.815669 (+ 2780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:11.815677 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:11.815689 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:11.816121 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:11.816498 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:11.816503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:11.816507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:11.816678 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:11.817214 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:11.817220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:11.817637 (+ 417us) client_negotiation.cc:769] Sending connection context
0619 04:38:11.817914 (+ 277us) client_negotiation.cc:240] Negotiation successful
0619 04:38:11.818408 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":277,"threads_started":1}
I20260619 04:38:11.820894 20181 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35551
I20260619 04:38:11.821173 20181 heartbeater.cc:464] Registering TS with master...
I20260619 04:38:11.821858 20181 heartbeater.cc:511] Master 127.0.97.62:35551 requested a full tablet report, sending...
I20260619 04:38:11.823256 19725 ts_manager.cc:194] Registered new tserver with Master: 283823fc897943aabb15cbf0b012e0fe (127.0.97.3:35267)
I20260619 04:38:11.824086 19725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56811
I20260619 04:38:11.831562 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:38:11.848526 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:11.836412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57054 (local address 127.0.97.62:35551)
0619 04:38:11.836617 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:11.836624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:11.837097 (+ 473us) server_negotiation.cc:407] Connection header received
0619 04:38:11.837260 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:11.837264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:11.837337 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:11.837439 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:11.838940 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:11.839645 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:11.840943 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:11.841390 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:11.841486 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:11.843661 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:11.843666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:11.843669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:11.843701 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:11.846174 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:11.846754 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:11.846757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:11.846759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:11.846869 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:11.847213 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:11.847216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:11.847218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:11.847500 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:11.847686 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:11.847998 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 04:38:11.848173 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 04:38:11.852227 19725 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:38:11.859022 19725 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:38:11.865252 19914 heartbeater.cc:503] Master 127.0.97.62:35551 was elected leader, sending a full tablet report...
I20260619 04:38:11.891932 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:11.871262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35267 (local address 127.0.0.1:33730)
0619 04:38:11.872009 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:11.872047 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:11.872216 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:11.883791 (+ 11575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:11.883805 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:11.883859 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:11.884394 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:11.884409 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:11.886928 (+ 2519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:11.886937 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:11.889529 (+ 2592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:11.889540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:11.890790 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:11.890797 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:11.891046 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:11.891105 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 04:38:11.891238 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 04:38:11.891461 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":32,"thread_start_us":1021,"threads_started":1}
I20260619 04:38:11.892889 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:11.872040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33730 (local address 127.0.97.3:35267)
0619 04:38:11.881589 (+ 9549us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:11.881599 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:11.881641 (+ 42us) server_negotiation.cc:407] Connection header received
0619 04:38:11.881761 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:11.881771 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:11.883280 (+ 1509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:11.884812 (+ 1532us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:11.885168 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:11.886777 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:11.889673 (+ 2896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:11.890591 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:11.890709 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:11.890969 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:11.892324 (+ 1355us) server_negotiation.cc:299] Negotiation successful
0619 04:38:11.892541 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9359,"thread_start_us":9201,"threads_started":1}
I20260619 04:38:11.893708 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:11.871388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34997 (local address 127.0.0.1:41644)
0619 04:38:11.884286 (+ 12898us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:11.884311 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:11.884564 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:11.886551 (+ 1987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:11.886556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:11.886585 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:11.886995 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:11.887005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:11.888817 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:11.888822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:11.890809 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:11.891049 (+ 240us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:11.892890 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:11.892896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:11.893055 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:11.893085 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 04:38:11.893181 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 04:38:11.893307 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12717,"mutex_wait_us":17,"thread_start_us":13166,"threads_started":1}
I20260619 04:38:11.894151 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:11.872584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.0.97.2:34997)
0619 04:38:11.885925 (+ 13341us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:11.885934 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:11.885961 (+ 27us) server_negotiation.cc:407] Connection header received
0619 04:38:11.886050 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:11.886057 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:11.886288 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:11.886658 (+ 370us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:11.887183 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:11.888698 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:11.891216 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:11.892703 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:11.892826 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:11.893027 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:11.893686 (+ 659us) server_negotiation.cc:299] Negotiation successful
0619 04:38:11.893844 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13174,"thread_start_us":13065,"threads_started":1}
I20260619 04:38:11.894196 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:11.871637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32997 (local address 127.0.0.1:38984)
0619 04:38:11.885582 (+ 13945us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:11.885603 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:11.885809 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:11.886427 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:11.886432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:11.886455 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:11.887495 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:11.887507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:11.889381 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:11.889386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:11.891476 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:11.891490 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:11.893426 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:11.893432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:11.893598 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:11.893628 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 04:38:11.893727 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 04:38:11.893861 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13818,"mutex_wait_us":4,"thread_start_us":13678,"threads_started":1}
I20260619 04:38:11.894889 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:11.872115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38984 (local address 127.0.97.1:32997)
0619 04:38:11.883733 (+ 11618us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:11.883743 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:11.885710 (+ 1967us) server_negotiation.cc:407] Connection header received
0619 04:38:11.886032 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:11.886039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:11.886245 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:11.886429 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:11.887678 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:11.889246 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:11.891659 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:11.893230 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:11.893360 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:11.893576 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:11.894406 (+ 830us) server_negotiation.cc:299] Negotiation successful
0619 04:38:11.894587 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11444,"thread_start_us":11127,"threads_started":1}
I20260619 04:38:11.897347 20116 tablet_service.cc:1467] Processing CreateTablet for tablet c37c91f07bc74f898e8f5bb2ff815bc7 (DEFAULT_TABLE table=test-table [id=4abcff8a21c64f37b249e3ef3e88ccb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:38:11.897727 19983 tablet_service.cc:1467] Processing CreateTablet for tablet c37c91f07bc74f898e8f5bb2ff815bc7 (DEFAULT_TABLE table=test-table [id=4abcff8a21c64f37b249e3ef3e88ccb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:38:11.897873 19849 tablet_service.cc:1467] Processing CreateTablet for tablet c37c91f07bc74f898e8f5bb2ff815bc7 (DEFAULT_TABLE table=test-table [id=4abcff8a21c64f37b249e3ef3e88ccb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:38:11.898913 20116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c37c91f07bc74f898e8f5bb2ff815bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:11.898947 19983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c37c91f07bc74f898e8f5bb2ff815bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:11.899063 19849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c37c91f07bc74f898e8f5bb2ff815bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:11.910588 20199 tablet_bootstrap.cc:492] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a: Bootstrap starting.
I20260619 04:38:11.912803 20200 tablet_bootstrap.cc:492] T c37c91f07bc74f898e8f5bb2ff815bc7 P 283823fc897943aabb15cbf0b012e0fe: Bootstrap starting.
I20260619 04:38:11.913612 20201 tablet_bootstrap.cc:492] T c37c91f07bc74f898e8f5bb2ff815bc7 P 8b56f0cb92d54f4a80333677b86ede02: Bootstrap starting.
I20260619 04:38:11.914687 20199 tablet_bootstrap.cc:654] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:11.916164 20199 log.cc:824] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:11.916633 20201 tablet_bootstrap.cc:654] T c37c91f07bc74f898e8f5bb2ff815bc7 P 8b56f0cb92d54f4a80333677b86ede02: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:11.916771 20200 tablet_bootstrap.cc:654] T c37c91f07bc74f898e8f5bb2ff815bc7 P 283823fc897943aabb15cbf0b012e0fe: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:11.917781 20201 log.cc:824] T c37c91f07bc74f898e8f5bb2ff815bc7 P 8b56f0cb92d54f4a80333677b86ede02: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:11.917866 20200 log.cc:824] T c37c91f07bc74f898e8f5bb2ff815bc7 P 283823fc897943aabb15cbf0b012e0fe: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:11.919520 20199 tablet_bootstrap.cc:492] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a: No bootstrap required, opened a new log
I20260619 04:38:11.919809 20199 ts_tablet_manager.cc:1397] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 04:38:11.920156 20201 tablet_bootstrap.cc:492] T c37c91f07bc74f898e8f5bb2ff815bc7 P 8b56f0cb92d54f4a80333677b86ede02: No bootstrap required, opened a new log
I20260619 04:38:11.920387 20201 ts_tablet_manager.cc:1397] T c37c91f07bc74f898e8f5bb2ff815bc7 P 8b56f0cb92d54f4a80333677b86ede02: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260619 04:38:11.925408 20200 tablet_bootstrap.cc:492] T c37c91f07bc74f898e8f5bb2ff815bc7 P 283823fc897943aabb15cbf0b012e0fe: No bootstrap required, opened a new log
I20260619 04:38:11.925688 20200 ts_tablet_manager.cc:1397] T c37c91f07bc74f898e8f5bb2ff815bc7 P 283823fc897943aabb15cbf0b012e0fe: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260619 04:38:11.928607 20199 raft_consensus.cc:348] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283823fc897943aabb15cbf0b012e0fe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35267 } } peers { permanent_uuid: "8b56f0cb92d54f4a80333677b86ede02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34997 } } peers { permanent_uuid: "b4cb80e9813e44b0bebbfba96385942a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32997 } }
I20260619 04:38:11.928952 20199 raft_consensus.cc:374] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:11.929080 20199 raft_consensus.cc:729] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4cb80e9813e44b0bebbfba96385942a, State: Initialized, Role: FOLLOWER
I20260619 04:38:11.929174 20201 raft_consensus.cc:348] T c37c91f07bc74f898e8f5bb2ff815bc7 P 8b56f0cb92d54f4a80333677b86ede02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283823fc897943aabb15cbf0b012e0fe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35267 } } peers { permanent_uuid: "8b56f0cb92d54f4a80333677b86ede02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34997 } } peers { permanent_uuid: "b4cb80e9813e44b0bebbfba96385942a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32997 } }
I20260619 04:38:11.929607 20201 raft_consensus.cc:374] T c37c91f07bc74f898e8f5bb2ff815bc7 P 8b56f0cb92d54f4a80333677b86ede02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:11.929700 20201 raft_consensus.cc:729] T c37c91f07bc74f898e8f5bb2ff815bc7 P 8b56f0cb92d54f4a80333677b86ede02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b56f0cb92d54f4a80333677b86ede02, State: Initialized, Role: FOLLOWER
I20260619 04:38:11.929664 20199 consensus_queue.cc:260] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283823fc897943aabb15cbf0b012e0fe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35267 } } peers { permanent_uuid: "8b56f0cb92d54f4a80333677b86ede02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34997 } } peers { permanent_uuid: "b4cb80e9813e44b0bebbfba96385942a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32997 } }
I20260619 04:38:11.930359 20201 consensus_queue.cc:260] T c37c91f07bc74f898e8f5bb2ff815bc7 P 8b56f0cb92d54f4a80333677b86ede02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283823fc897943aabb15cbf0b012e0fe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35267 } } peers { permanent_uuid: "8b56f0cb92d54f4a80333677b86ede02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34997 } } peers { permanent_uuid: "b4cb80e9813e44b0bebbfba96385942a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32997 } }
I20260619 04:38:11.931340 20199 ts_tablet_manager.cc:1428] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260619 04:38:11.932262 20048 heartbeater.cc:503] Master 127.0.97.62:35551 was elected leader, sending a full tablet report...
I20260619 04:38:11.932976 20201 ts_tablet_manager.cc:1428] T c37c91f07bc74f898e8f5bb2ff815bc7 P 8b56f0cb92d54f4a80333677b86ede02: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260619 04:38:11.937021 20200 raft_consensus.cc:348] T c37c91f07bc74f898e8f5bb2ff815bc7 P 283823fc897943aabb15cbf0b012e0fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283823fc897943aabb15cbf0b012e0fe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35267 } } peers { permanent_uuid: "8b56f0cb92d54f4a80333677b86ede02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34997 } } peers { permanent_uuid: "b4cb80e9813e44b0bebbfba96385942a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32997 } }
I20260619 04:38:11.937356 20200 raft_consensus.cc:374] T c37c91f07bc74f898e8f5bb2ff815bc7 P 283823fc897943aabb15cbf0b012e0fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:11.937425 20200 raft_consensus.cc:729] T c37c91f07bc74f898e8f5bb2ff815bc7 P 283823fc897943aabb15cbf0b012e0fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283823fc897943aabb15cbf0b012e0fe, State: Initialized, Role: FOLLOWER
I20260619 04:38:11.937875 20200 consensus_queue.cc:260] T c37c91f07bc74f898e8f5bb2ff815bc7 P 283823fc897943aabb15cbf0b012e0fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283823fc897943aabb15cbf0b012e0fe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35267 } } peers { permanent_uuid: "8b56f0cb92d54f4a80333677b86ede02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34997 } } peers { permanent_uuid: "b4cb80e9813e44b0bebbfba96385942a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32997 } }
I20260619 04:38:11.938773 20181 heartbeater.cc:503] Master 127.0.97.62:35551 was elected leader, sending a full tablet report...
I20260619 04:38:11.939159 20200 ts_tablet_manager.cc:1428] T c37c91f07bc74f898e8f5bb2ff815bc7 P 283823fc897943aabb15cbf0b012e0fe: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260619 04:38:12.051069 20182 tablet.cc:2369] T c37c91f07bc74f898e8f5bb2ff815bc7 P 283823fc897943aabb15cbf0b012e0fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:38:12.071826 20049 tablet.cc:2369] T c37c91f07bc74f898e8f5bb2ff815bc7 P 8b56f0cb92d54f4a80333677b86ede02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:38:12.074133 20205 raft_consensus.cc:482] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:38:12.074457 20205 raft_consensus.cc:504] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283823fc897943aabb15cbf0b012e0fe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35267 } } peers { permanent_uuid: "8b56f0cb92d54f4a80333677b86ede02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34997 } } peers { permanent_uuid: "b4cb80e9813e44b0bebbfba96385942a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32997 } }
I20260619 04:38:12.076174 20205 leader_election.cc:288] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 283823fc897943aabb15cbf0b012e0fe (127.0.97.3:35267), 8b56f0cb92d54f4a80333677b86ede02 (127.0.97.2:34997)
I20260619 04:38:12.082573 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:12.076685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45627 (local address 127.0.97.3:35267)
0619 04:38:12.076890 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:12.076897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:12.077536 (+ 639us) server_negotiation.cc:407] Connection header received
0619 04:38:12.077689 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:12.077692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:12.077751 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:12.077831 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:12.078383 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:12.079102 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:12.080844 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:12.081611 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:12.081721 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:12.081835 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:12.082187 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 04:38:12.082315 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 04:38:12.082581 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:12.076624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53185 (local address 127.0.97.2:34997)
0619 04:38:12.076829 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:12.076837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:12.077464 (+ 627us) server_negotiation.cc:407] Connection header received
0619 04:38:12.077649 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:12.077653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:12.077714 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:12.077785 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:12.078369 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:12.079056 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:12.080846 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:12.081649 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:12.081763 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:12.081853 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:12.082192 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 04:38:12.082351 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 04:38:12.082757 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:12.076551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35267 (local address 127.0.97.1:45627)
0619 04:38:12.077432 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:12.077447 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:12.077567 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:12.077845 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:12.077849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:12.077870 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:12.078237 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:12.078245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:12.079228 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:12.079232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:12.080702 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:12.080713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:12.081816 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:12.081822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:12.081965 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:12.082064 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 04:38:12.082145 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 04:38:12.082315 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":91,"thread_start_us":525,"threads_started":1}
I20260619 04:38:12.082864 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:12.076535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34997 (local address 127.0.97.1:53185)
0619 04:38:12.077364 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:12.077379 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:12.077522 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:12.077817 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:12.077821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:12.077843 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:12.078235 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:12.078244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:12.079171 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:12.079177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:12.080707 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:12.080718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:12.081858 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:12.081865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:12.082055 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:12.082079 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 04:38:12.082152 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 04:38:12.082304 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":52,"thread_start_us":484,"threads_started":1}
I20260619 04:38:12.083818 20003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37c91f07bc74f898e8f5bb2ff815bc7" candidate_uuid: "b4cb80e9813e44b0bebbfba96385942a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b56f0cb92d54f4a80333677b86ede02" is_pre_election: true
I20260619 04:38:12.084189 20136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37c91f07bc74f898e8f5bb2ff815bc7" candidate_uuid: "b4cb80e9813e44b0bebbfba96385942a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283823fc897943aabb15cbf0b012e0fe" is_pre_election: true
I20260619 04:38:12.084306 20003 raft_consensus.cc:2454] T c37c91f07bc74f898e8f5bb2ff815bc7 P 8b56f0cb92d54f4a80333677b86ede02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4cb80e9813e44b0bebbfba96385942a in term 0.
I20260619 04:38:12.084648 20136 raft_consensus.cc:2454] T c37c91f07bc74f898e8f5bb2ff815bc7 P 283823fc897943aabb15cbf0b012e0fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4cb80e9813e44b0bebbfba96385942a in term 0.
I20260619 04:38:12.084962 19802 leader_election.cc:302] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b56f0cb92d54f4a80333677b86ede02, b4cb80e9813e44b0bebbfba96385942a; no voters:
I20260619 04:38:12.085402 20205 raft_consensus.cc:2781] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:38:12.085551 20205 raft_consensus.cc:482] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:38:12.085645 20205 raft_consensus.cc:3037] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a [term 0 FOLLOWER]: Advancing to term 1
W20260619 04:38:12.085898 19915 tablet.cc:2369] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:38:12.087920 20205 raft_consensus.cc:504] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283823fc897943aabb15cbf0b012e0fe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35267 } } peers { permanent_uuid: "8b56f0cb92d54f4a80333677b86ede02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34997 } } peers { permanent_uuid: "b4cb80e9813e44b0bebbfba96385942a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32997 } }
I20260619 04:38:12.088573 20205 leader_election.cc:288] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a [CANDIDATE]: Term 1 election: Requested vote from peers 283823fc897943aabb15cbf0b012e0fe (127.0.97.3:35267), 8b56f0cb92d54f4a80333677b86ede02 (127.0.97.2:34997)
I20260619 04:38:12.089032 20136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37c91f07bc74f898e8f5bb2ff815bc7" candidate_uuid: "b4cb80e9813e44b0bebbfba96385942a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283823fc897943aabb15cbf0b012e0fe"
I20260619 04:38:12.089102 20003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37c91f07bc74f898e8f5bb2ff815bc7" candidate_uuid: "b4cb80e9813e44b0bebbfba96385942a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b56f0cb92d54f4a80333677b86ede02"
I20260619 04:38:12.089259 20136 raft_consensus.cc:3037] T c37c91f07bc74f898e8f5bb2ff815bc7 P 283823fc897943aabb15cbf0b012e0fe [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:12.089315 20003 raft_consensus.cc:3037] T c37c91f07bc74f898e8f5bb2ff815bc7 P 8b56f0cb92d54f4a80333677b86ede02 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:12.092417 20003 raft_consensus.cc:2454] T c37c91f07bc74f898e8f5bb2ff815bc7 P 8b56f0cb92d54f4a80333677b86ede02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4cb80e9813e44b0bebbfba96385942a in term 1.
I20260619 04:38:12.092419 20136 raft_consensus.cc:2454] T c37c91f07bc74f898e8f5bb2ff815bc7 P 283823fc897943aabb15cbf0b012e0fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4cb80e9813e44b0bebbfba96385942a in term 1.
I20260619 04:38:12.092860 19802 leader_election.cc:302] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b56f0cb92d54f4a80333677b86ede02, b4cb80e9813e44b0bebbfba96385942a; no voters:
I20260619 04:38:12.093117 20205 raft_consensus.cc:2781] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:12.093566 20205 raft_consensus.cc:686] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a [term 1 LEADER]: Becoming Leader. State: Replica: b4cb80e9813e44b0bebbfba96385942a, State: Running, Role: LEADER
I20260619 04:38:12.094007 20205 consensus_queue.cc:237] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283823fc897943aabb15cbf0b012e0fe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35267 } } peers { permanent_uuid: "8b56f0cb92d54f4a80333677b86ede02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34997 } } peers { permanent_uuid: "b4cb80e9813e44b0bebbfba96385942a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32997 } }
I20260619 04:38:12.098660 19724 catalog_manager.cc:5495] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a reported cstate change: term changed from 0 to 1, leader changed from <none> to b4cb80e9813e44b0bebbfba96385942a (127.0.97.1). New cstate: current_term: 1 leader_uuid: "b4cb80e9813e44b0bebbfba96385942a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283823fc897943aabb15cbf0b012e0fe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b56f0cb92d54f4a80333677b86ede02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4cb80e9813e44b0bebbfba96385942a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32997 } health_report { overall_health: HEALTHY } } }
I20260619 04:38:12.131313 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:12.126757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39000 (local address 127.0.97.1:32997)
0619 04:38:12.126932 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:12.126937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:12.126953 (+ 16us) server_negotiation.cc:407] Connection header received
0619 04:38:12.127037 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:12.127041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:12.127139 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:12.127265 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:12.127558 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:12.128247 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:12.129550 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:12.129944 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:12.130032 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:12.130414 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:12.130905 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:12.131012 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:12.131055 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 04:38:12.131118 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 04:38:12.139211 20136 raft_consensus.cc:1261] T c37c91f07bc74f898e8f5bb2ff815bc7 P 283823fc897943aabb15cbf0b012e0fe [term 1 FOLLOWER]: Refusing update from remote peer b4cb80e9813e44b0bebbfba96385942a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:38:12.139211 20003 raft_consensus.cc:1261] T c37c91f07bc74f898e8f5bb2ff815bc7 P 8b56f0cb92d54f4a80333677b86ede02 [term 1 FOLLOWER]: Refusing update from remote peer b4cb80e9813e44b0bebbfba96385942a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:38:12.140117 20205 consensus_queue.cc:1035] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b56f0cb92d54f4a80333677b86ede02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:38:12.140404 20210 consensus_queue.cc:1035] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a [LEADER]: Connected to new peer: Peer: permanent_uuid: "283823fc897943aabb15cbf0b012e0fe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:38:12.150986 20213 mvcc.cc:205] Tried to move back new op lower bound from 7298432582192914432 to 7298432582020546560. Current Snapshot: MvccSnapshot[applied={T|T < 7298432582192914432}]
I20260619 04:38:12.155925 20214 mvcc.cc:205] Tried to move back new op lower bound from 7298432582192914432 to 7298432582020546560. Current Snapshot: MvccSnapshot[applied={T|T < 7298432582192914432 or (T in {7298432582192914432})}]
I20260619 04:38:12.166828 19724 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:38:12.167121 19724 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:38:12.170642 19724 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95553cfdb32046e09e30948e539aa464: Sending DeleteTablet for 3 replicas of tablet c37c91f07bc74f898e8f5bb2ff815bc7
I20260619 04:38:12.171744 20116 tablet_service.cc:1514] Processing DeleteTablet for tablet c37c91f07bc74f898e8f5bb2ff815bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:38:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33730
I20260619 04:38:12.171751 19849 tablet_service.cc:1514] Processing DeleteTablet for tablet c37c91f07bc74f898e8f5bb2ff815bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:38:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38984
I20260619 04:38:12.171774 19983 tablet_service.cc:1514] Processing DeleteTablet for tablet c37c91f07bc74f898e8f5bb2ff815bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:38:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41644
I20260619 04:38:12.176280 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 19786
I20260619 04:38:12.187031 20225 tablet_replica.cc:330] T c37c91f07bc74f898e8f5bb2ff815bc7 P 8b56f0cb92d54f4a80333677b86ede02: stopping tablet replica
I20260619 04:38:12.187706 20225 raft_consensus.cc:2229] T c37c91f07bc74f898e8f5bb2ff815bc7 P 8b56f0cb92d54f4a80333677b86ede02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:38:12.188400 20225 raft_consensus.cc:2258] T c37c91f07bc74f898e8f5bb2ff815bc7 P 8b56f0cb92d54f4a80333677b86ede02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:38:12.190135 20223 tablet_replica.cc:330] T c37c91f07bc74f898e8f5bb2ff815bc7 P 283823fc897943aabb15cbf0b012e0fe: stopping tablet replica
I20260619 04:38:12.190547 20223 raft_consensus.cc:2229] T c37c91f07bc74f898e8f5bb2ff815bc7 P 283823fc897943aabb15cbf0b012e0fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:38:12.191021 20223 raft_consensus.cc:2258] T c37c91f07bc74f898e8f5bb2ff815bc7 P 283823fc897943aabb15cbf0b012e0fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:38:12.193181 20225 ts_tablet_manager.cc:1905] T c37c91f07bc74f898e8f5bb2ff815bc7 P 8b56f0cb92d54f4a80333677b86ede02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:38:12.193238 20223 ts_tablet_manager.cc:1905] T c37c91f07bc74f898e8f5bb2ff815bc7 P 283823fc897943aabb15cbf0b012e0fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:38:12.194437 20224 tablet_replica.cc:330] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a: stopping tablet replica
I20260619 04:38:12.194875 20224 raft_consensus.cc:2229] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:38:12.195647 20224 raft_consensus.cc:2258] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:38:12.200240 20225 ts_tablet_manager.cc:1918] T c37c91f07bc74f898e8f5bb2ff815bc7 P 8b56f0cb92d54f4a80333677b86ede02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:38:12.200500 20225 log.cc:1196] T c37c91f07bc74f898e8f5bb2ff815bc7 P 8b56f0cb92d54f4a80333677b86ede02: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/c37c91f07bc74f898e8f5bb2ff815bc7
Jun 19 04:38:12 dist-test-slave-n3tq krb5kdc[19680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 04:38:12.201048 20225 ts_tablet_manager.cc:1939] T c37c91f07bc74f898e8f5bb2ff815bc7 P 8b56f0cb92d54f4a80333677b86ede02: Deleting consensus metadata
I20260619 04:38:12.201830 20223 ts_tablet_manager.cc:1918] T c37c91f07bc74f898e8f5bb2ff815bc7 P 283823fc897943aabb15cbf0b012e0fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:38:12.202046 20224 ts_tablet_manager.cc:1905] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:38:12.203154 19709 catalog_manager.cc:4841] TS 8b56f0cb92d54f4a80333677b86ede02 (127.0.97.2:34997): tablet c37c91f07bc74f898e8f5bb2ff815bc7 (table test-table [id=4abcff8a21c64f37b249e3ef3e88ccb8]) successfully deleted
I20260619 04:38:12.203670 20223 log.cc:1196] T c37c91f07bc74f898e8f5bb2ff815bc7 P 283823fc897943aabb15cbf0b012e0fe: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/c37c91f07bc74f898e8f5bb2ff815bc7
I20260619 04:38:12.204540 20223 ts_tablet_manager.cc:1939] T c37c91f07bc74f898e8f5bb2ff815bc7 P 283823fc897943aabb15cbf0b012e0fe: Deleting consensus metadata
I20260619 04:38:12.206241 19712 catalog_manager.cc:4841] TS 283823fc897943aabb15cbf0b012e0fe (127.0.97.3:35267): tablet c37c91f07bc74f898e8f5bb2ff815bc7 (table test-table [id=4abcff8a21c64f37b249e3ef3e88ccb8]) successfully deleted
I20260619 04:38:12.208362 20224 ts_tablet_manager.cc:1918] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:38:12.208549 20224 log.cc:1196] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781843588881932-388-0/minicluster-data/ts-0/wal/wals/c37c91f07bc74f898e8f5bb2ff815bc7
I20260619 04:38:12.208941 20224 ts_tablet_manager.cc:1939] T c37c91f07bc74f898e8f5bb2ff815bc7 P b4cb80e9813e44b0bebbfba96385942a: Deleting consensus metadata
I20260619 04:38:12.209709 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:12.177448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39008 (local address 127.0.97.1:32997)
0619 04:38:12.177630 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:12.177636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:12.189502 (+ 11866us) server_negotiation.cc:407] Connection header received
0619 04:38:12.189708 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:12.189714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:12.189822 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:12.189916 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:12.191289 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:12.193582 (+ 2293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:12.195351 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:12.196038 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:12.196160 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:12.202015 (+ 5855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:12.202035 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:12.202052 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:12.202093 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:12.206168 (+ 4075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:12.207023 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:12.207035 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:12.207039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:12.207239 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:12.207720 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:12.207725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:12.207728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:12.208169 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:12.208472 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:12.209073 (+ 601us) server_negotiation.cc:299] Negotiation successful
0619 04:38:12.209325 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 04:38:12.210281 19711 catalog_manager.cc:4841] TS b4cb80e9813e44b0bebbfba96385942a (127.0.97.1:32997): tablet c37c91f07bc74f898e8f5bb2ff815bc7 (table test-table [id=4abcff8a21c64f37b249e3ef3e88ccb8]) successfully deleted
I20260619 04:38:12.211050 19909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:12.310571 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 19786
I20260619 04:38:12.325948 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 19920
Jun 19 04:38:12 dist-test-slave-n3tq krb5kdc[19680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 04:38:12.342938 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:12.326688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41648 (local address 127.0.97.2:34997)
0619 04:38:12.326867 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:12.326871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:12.326925 (+ 54us) server_negotiation.cc:407] Connection header received
0619 04:38:12.327089 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:12.327092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:12.327152 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:12.327233 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:12.328844 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:12.329609 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:12.331448 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:12.331872 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:12.331984 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:12.337082 (+ 5098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:12.337094 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:12.337108 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:12.337144 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:12.339942 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:12.340814 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:12.340824 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:12.340828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:12.341008 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:12.341505 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:12.341510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:12.341512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:12.341906 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:12.342154 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:12.342425 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 04:38:12.342614 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:38:12.343518 20043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:12.432382 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 19920
I20260619 04:38:12.446221 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 20053
Jun 19 04:38:12 dist-test-slave-n3tq krb5kdc[19680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 04:38:12.460939 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:12.446895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33744 (local address 127.0.97.3:35267)
0619 04:38:12.447053 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:12.447058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:12.447076 (+ 18us) server_negotiation.cc:407] Connection header received
0619 04:38:12.447217 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:12.447221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:12.447284 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:12.447409 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:12.448644 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:12.449362 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:12.450624 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:12.451019 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:12.451122 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:12.455534 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:12.455550 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:12.455567 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:12.455605 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:12.458410 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:12.459175 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:12.459183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:12.459185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:12.459364 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:12.459658 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:12.459661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:12.459663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:12.459998 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:12.460211 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:12.460502 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 04:38:12.460669 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 04:38:12.461583 20176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:12.548996 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 20053
I20260619 04:38:12.562014 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 19693
I20260619 04:38:12.563051 19755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:12.676157 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 19693
2026-06-19T04:38:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4099 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T04:38:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:12Z Disabled control of system clock
I20260619 04:38:12.744668 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38813
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 04:38:13.057935 20234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:13.058219 20234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:13.058323 20234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:13.065511 20234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:13.065614 20234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:13.065654 20234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:13.065739 20234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:13.065801 20234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:13.075012 20234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:13.076311 20234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:13.078080 20234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:13.089272 20239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:13.089366 20242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:13.089236 20240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:13.090196 20234 server_base.cc:1034] running on GCE node
I20260619 04:38:13.090906 20234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:13.092235 20234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:13.093436 20234 hybrid_clock.cc:648] HybridClock initialized: now 1781843893093377 us; error 48 us; skew 500 ppm
I20260619 04:38:13.093756 20234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:38:13.095922 20234 webserver.cc:460] Webserver started at http://127.0.97.62:38491/ using document root <none> and password file <none>
I20260619 04:38:13.096581 20234 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:13.096665 20234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:13.096925 20234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:13.099687 20234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "3c65d6b1c2d44b3faff72c97d6254c34"
format_stamp: "Formatted at 2026-06-19 04:38:13 on dist-test-slave-n3tq"
server_key: "9ff85543bfc06c43db43441d6ca72135"
server_key_iv: "af1b340711fa8c7598f45dc42d0e18b4"
server_key_version: "encryptionkey@0"
I20260619 04:38:13.100410 20234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "3c65d6b1c2d44b3faff72c97d6254c34"
format_stamp: "Formatted at 2026-06-19 04:38:13 on dist-test-slave-n3tq"
server_key: "9ff85543bfc06c43db43441d6ca72135"
server_key_iv: "af1b340711fa8c7598f45dc42d0e18b4"
server_key_version: "encryptionkey@0"
I20260619 04:38:13.105240 20234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 04:38:13.108649 20248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:13.109977 20234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:38:13.110136 20234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "3c65d6b1c2d44b3faff72c97d6254c34"
format_stamp: "Formatted at 2026-06-19 04:38:13 on dist-test-slave-n3tq"
server_key: "9ff85543bfc06c43db43441d6ca72135"
server_key_iv: "af1b340711fa8c7598f45dc42d0e18b4"
server_key_version: "encryptionkey@0"
I20260619 04:38:13.110328 20234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:13.130028 20234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:13.130923 20234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:13.131193 20234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:13.148407 20234 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38813
I20260619 04:38:13.148427 20299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38813 every 8 connection(s)
I20260619 04:38:13.149755 20234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:13.153406 20300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:13.155772 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 20234
I20260619 04:38:13.155962 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:13.156508 388 external_mini_cluster.cc:1428] Setting key b5d27f6995ea4669f1696e37468d0b1f
I20260619 04:38:13.164883 20300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34: Bootstrap starting.
I20260619 04:38:13.167951 20300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:13.169201 20300 log.cc:824] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:13.169786 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:13.158541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50510 (local address 127.0.97.62:38813)
0619 04:38:13.159539 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:13.159550 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:13.159593 (+ 43us) server_negotiation.cc:407] Connection header received
0619 04:38:13.161272 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:13.161295 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:13.161773 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:13.162124 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:13.162686 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:13.163931 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:13.165301 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:13.165942 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:13.166111 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:13.167210 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:13.167218 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:13.167230 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:13.167735 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:13.167819 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:13.167839 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:13.167952 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:13.168180 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 04:38:13.168544 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":348,"threads_started":1}
I20260619 04:38:13.171974 20300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34: No bootstrap required, opened a new log
I20260619 04:38:13.178870 20300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c65d6b1c2d44b3faff72c97d6254c34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38813 } }
I20260619 04:38:13.179196 20300 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:13.179297 20300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c65d6b1c2d44b3faff72c97d6254c34, State: Initialized, Role: FOLLOWER
I20260619 04:38:13.179816 20300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c65d6b1c2d44b3faff72c97d6254c34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38813 } }
I20260619 04:38:13.180018 20300 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:38:13.180137 20300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:38:13.180274 20300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:13.182404 20300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c65d6b1c2d44b3faff72c97d6254c34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38813 } }
I20260619 04:38:13.182883 20300 leader_election.cc:302] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c65d6b1c2d44b3faff72c97d6254c34; no voters:
I20260619 04:38:13.183393 20300 leader_election.cc:288] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:38:13.183521 20305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:13.184134 20305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34 [term 1 LEADER]: Becoming Leader. State: Replica: 3c65d6b1c2d44b3faff72c97d6254c34, State: Running, Role: LEADER
I20260619 04:38:13.184639 20305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c65d6b1c2d44b3faff72c97d6254c34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38813 } }
I20260619 04:38:13.185353 20300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:13.187171 20307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c65d6b1c2d44b3faff72c97d6254c34. Latest consensus state: current_term: 1 leader_uuid: "3c65d6b1c2d44b3faff72c97d6254c34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c65d6b1c2d44b3faff72c97d6254c34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38813 } } }
I20260619 04:38:13.187556 20307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:13.189461 20312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:13.189270 20306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c65d6b1c2d44b3faff72c97d6254c34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c65d6b1c2d44b3faff72c97d6254c34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38813 } } }
I20260619 04:38:13.189661 20306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:13.194772 20312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:13.203913 20312 catalog_manager.cc:1347] Generated new cluster ID: f7572fdae6f44942b30815c079cbea80
I20260619 04:38:13.204010 20312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:13.229455 20312 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:13.230715 20312 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:13.240165 20312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34: Generated new TSK 0
I20260619 04:38:13.241096 20312 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:38:13.253660 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38813
--builtin_ntp_servers=127.0.97.20:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 04:38:13.595558 20324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:13.595839 20324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:13.595921 20324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:13.603206 20324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:13.603322 20324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:13.603430 20324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 04:38:13.612597 20324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:13.613863 20324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:13.615707 20324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:13.628000 20330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:13.628008 20332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:13.628207 20329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:13.628841 20324 server_base.cc:1034] running on GCE node
I20260619 04:38:13.629396 20324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:13.630277 20324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:13.631479 20324 hybrid_clock.cc:648] HybridClock initialized: now 1781843893631419 us; error 52 us; skew 500 ppm
I20260619 04:38:13.631793 20324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:38:13.633919 20324 webserver.cc:460] Webserver started at http://127.0.97.1:46201/ using document root <none> and password file <none>
I20260619 04:38:13.634684 20324 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:13.634797 20324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:13.635092 20324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:13.637912 20324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-0/data/instance:
uuid: "6b79b3ec3f694980a4ced07c8c1f76d0"
format_stamp: "Formatted at 2026-06-19 04:38:13 on dist-test-slave-n3tq"
server_key: "21edee11041d77fd999fa4f730fe5aa7"
server_key_iv: "2291fb63083a9afa589d4e67be203552"
server_key_version: "encryptionkey@0"
I20260619 04:38:13.638648 20324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6b79b3ec3f694980a4ced07c8c1f76d0"
format_stamp: "Formatted at 2026-06-19 04:38:13 on dist-test-slave-n3tq"
server_key: "21edee11041d77fd999fa4f730fe5aa7"
server_key_iv: "2291fb63083a9afa589d4e67be203552"
server_key_version: "encryptionkey@0"
I20260619 04:38:13.643561 20324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 04:38:13.646930 20338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:13.648201 20324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:38:13.648353 20324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-0/wal
uuid: "6b79b3ec3f694980a4ced07c8c1f76d0"
format_stamp: "Formatted at 2026-06-19 04:38:13 on dist-test-slave-n3tq"
server_key: "21edee11041d77fd999fa4f730fe5aa7"
server_key_iv: "2291fb63083a9afa589d4e67be203552"
server_key_version: "encryptionkey@0"
I20260619 04:38:13.648607 20324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:13.664268 20324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:13.665164 20324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:13.665457 20324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:13.666424 20324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:38:13.667907 20324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:38:13.667999 20324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:13.668143 20324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:38:13.668212 20324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:13.697562 20324 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35601
I20260619 04:38:13.697606 20450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35601 every 8 connection(s)
I20260619 04:38:13.698803 20324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-0/data/info.pb
I20260619 04:38:13.700217 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 20324
I20260619 04:38:13.700380 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-0/wal/instance
I20260619 04:38:13.700914 388 external_mini_cluster.cc:1428] Setting key 0bc7c43b2e375dd7b3b58edd1ad4708d
I20260619 04:38:13.703951 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38813
--builtin_ntp_servers=127.0.97.20:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 04:38:13.713321 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:13.702018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38519 (local address 127.0.97.62:38813)
0619 04:38:13.702469 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:13.702475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:13.703759 (+ 1284us) server_negotiation.cc:407] Connection header received
0619 04:38:13.706182 (+ 2423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:13.706187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:13.706314 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:13.706406 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:13.707716 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:13.708427 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:13.710590 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:13.711003 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:13.711101 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:13.712606 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:13.712610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:13.712614 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:13.712633 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:13.712673 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:13.712685 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:13.712760 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:13.712970 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 04:38:13.713049 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":233,"threads_started":1}
I20260619 04:38:13.714689 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:13.702517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38813 (local address 127.0.97.1:38519)
0619 04:38:13.703586 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:13.703639 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:13.705865 (+ 2226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:13.706665 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:13.706677 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:13.706803 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:13.707544 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:13.707565 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:13.708597 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:13.708602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:13.710446 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:13.710459 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:13.711207 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:13.711213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:13.711477 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:13.712252 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:38:13.712254 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:13.712295 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:38:13.712378 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:13.712770 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:13.712781 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:13.712788 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 04:38:13.712969 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 04:38:13.713401 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":320,"threads_started":1}
I20260619 04:38:13.716178 20451 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38813
I20260619 04:38:13.716480 20451 heartbeater.cc:464] Registering TS with master...
I20260619 04:38:13.717240 20451 heartbeater.cc:511] Master 127.0.97.62:38813 requested a full tablet report, sending...
I20260619 04:38:13.719197 20265 ts_manager.cc:194] Registered new tserver with Master: 6b79b3ec3f694980a4ced07c8c1f76d0 (127.0.97.1:35601)
I20260619 04:38:13.720683 20265 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:38519
W20260619 04:38:14.022004 20455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:14.022275 20455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:14.022383 20455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:14.029639 20455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:14.029752 20455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:14.029893 20455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 04:38:14.038971 20455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:14.040225 20455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:14.041949 20455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:14.054412 20460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:14.054391 20463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:14.054438 20461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:14.055094 20455 server_base.cc:1034] running on GCE node
I20260619 04:38:14.055681 20455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:14.056631 20455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:14.057830 20455 hybrid_clock.cc:648] HybridClock initialized: now 1781843894057775 us; error 42 us; skew 500 ppm
I20260619 04:38:14.058130 20455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:38:14.060418 20455 webserver.cc:460] Webserver started at http://127.0.97.2:36151/ using document root <none> and password file <none>
I20260619 04:38:14.061142 20455 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:14.061229 20455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:14.061555 20455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:14.064370 20455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-1/data/instance:
uuid: "4f855a4f98204b348eb3f067527aadcb"
format_stamp: "Formatted at 2026-06-19 04:38:14 on dist-test-slave-n3tq"
server_key: "89b3500aec5b14294112dc79a10cedeb"
server_key_iv: "6b8392f7bc7293a7e118874c7bdfb043"
server_key_version: "encryptionkey@0"
I20260619 04:38:14.065168 20455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4f855a4f98204b348eb3f067527aadcb"
format_stamp: "Formatted at 2026-06-19 04:38:14 on dist-test-slave-n3tq"
server_key: "89b3500aec5b14294112dc79a10cedeb"
server_key_iv: "6b8392f7bc7293a7e118874c7bdfb043"
server_key_version: "encryptionkey@0"
I20260619 04:38:14.070194 20455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 04:38:14.073688 20469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:14.075042 20455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 04:38:14.075187 20455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-1/wal
uuid: "4f855a4f98204b348eb3f067527aadcb"
format_stamp: "Formatted at 2026-06-19 04:38:14 on dist-test-slave-n3tq"
server_key: "89b3500aec5b14294112dc79a10cedeb"
server_key_iv: "6b8392f7bc7293a7e118874c7bdfb043"
server_key_version: "encryptionkey@0"
I20260619 04:38:14.075384 20455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:14.101470 20455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:14.102330 20455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:14.102589 20455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:14.103607 20455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:38:14.105140 20455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:38:14.105233 20455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:14.105342 20455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:38:14.105414 20455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:14.135324 20455 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41837
I20260619 04:38:14.135363 20581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41837 every 8 connection(s)
I20260619 04:38:14.136603 20455 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-1/data/info.pb
I20260619 04:38:14.140419 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 20455
I20260619 04:38:14.140617 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-1/wal/instance
I20260619 04:38:14.141110 388 external_mini_cluster.cc:1428] Setting key a3997a20c6713e036b38f6538b26c7c1
I20260619 04:38:14.143747 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38813
--builtin_ntp_servers=127.0.97.20:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 04:38:14.149478 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:14.139140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58491 (local address 127.0.97.62:38813)
0619 04:38:14.139346 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:14.139351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:14.140873 (+ 1522us) server_negotiation.cc:407] Connection header received
0619 04:38:14.142856 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:14.142862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:14.142940 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:14.143029 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:14.144135 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.144936 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.146650 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.147082 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.147201 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:14.148656 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:14.148663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:14.148667 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:14.148696 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:14.148737 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:14.148754 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:14.148856 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:14.149090 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 04:38:14.149192 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 04:38:14.150770 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:14.139645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38813 (local address 127.0.97.2:58491)
0619 04:38:14.140712 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:14.140762 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:14.142544 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:14.143260 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:14.143269 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:14.143367 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:14.143985 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:14.144005 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.145072 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.145077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:14.146520 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:14.146532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.147314 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.147322 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:14.147529 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:14.148259 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:38:14.148263 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:14.148316 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:38:14.148431 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:14.148872 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:14.148882 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:14.148890 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 04:38:14.149071 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 04:38:14.149487 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":313,"threads_started":1}
I20260619 04:38:14.153029 20582 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38813
I20260619 04:38:14.153355 20582 heartbeater.cc:464] Registering TS with master...
I20260619 04:38:14.154096 20582 heartbeater.cc:511] Master 127.0.97.62:38813 requested a full tablet report, sending...
I20260619 04:38:14.155604 20265 ts_manager.cc:194] Registered new tserver with Master: 4f855a4f98204b348eb3f067527aadcb (127.0.97.2:41837)
I20260619 04:38:14.156785 20265 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:58491
W20260619 04:38:14.464771 20585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:14.465073 20585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:14.465197 20585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:14.472316 20585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:14.472432 20585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:14.472591 20585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 04:38:14.481851 20585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:14.483170 20585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:14.484963 20585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:14.498176 20591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:14.498253 20593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:14.498219 20590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:14.498575 20585 server_base.cc:1034] running on GCE node
I20260619 04:38:14.499457 20585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:14.500370 20585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:14.501598 20585 hybrid_clock.cc:648] HybridClock initialized: now 1781843894501542 us; error 33 us; skew 500 ppm
I20260619 04:38:14.501931 20585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:38:14.504208 20585 webserver.cc:460] Webserver started at http://127.0.97.3:33875/ using document root <none> and password file <none>
I20260619 04:38:14.504904 20585 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:14.504995 20585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:14.505311 20585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:14.508037 20585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-2/data/instance:
uuid: "39444f08a2534d6ea56faa6c0e524760"
format_stamp: "Formatted at 2026-06-19 04:38:14 on dist-test-slave-n3tq"
server_key: "3296b527eaf27d3b2abc93aa3676ea40"
server_key_iv: "b6fe50bbac1d94fe899223cc7f8ed8ad"
server_key_version: "encryptionkey@0"
I20260619 04:38:14.508798 20585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "39444f08a2534d6ea56faa6c0e524760"
format_stamp: "Formatted at 2026-06-19 04:38:14 on dist-test-slave-n3tq"
server_key: "3296b527eaf27d3b2abc93aa3676ea40"
server_key_iv: "b6fe50bbac1d94fe899223cc7f8ed8ad"
server_key_version: "encryptionkey@0"
I20260619 04:38:14.513538 20585 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 04:38:14.516913 20599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:14.518160 20585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 04:38:14.518301 20585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-2/wal
uuid: "39444f08a2534d6ea56faa6c0e524760"
format_stamp: "Formatted at 2026-06-19 04:38:14 on dist-test-slave-n3tq"
server_key: "3296b527eaf27d3b2abc93aa3676ea40"
server_key_iv: "b6fe50bbac1d94fe899223cc7f8ed8ad"
server_key_version: "encryptionkey@0"
I20260619 04:38:14.518486 20585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:14.537619 20585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:14.538430 20585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:14.538686 20585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:14.539726 20585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:38:14.541266 20585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:38:14.541363 20585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:14.541468 20585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:38:14.541539 20585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:14.570092 20585 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36623
I20260619 04:38:14.570170 20711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36623 every 8 connection(s)
I20260619 04:38:14.571350 20585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-2/data/info.pb
I20260619 04:38:14.580443 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 20585
I20260619 04:38:14.580627 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-2/wal/instance
I20260619 04:38:14.581130 388 external_mini_cluster.cc:1428] Setting key 18bc9f0dc0d857110096b9801c5cc06a
I20260619 04:38:14.583318 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:14.573746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33589 (local address 127.0.97.62:38813)
0619 04:38:14.573916 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:14.573920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:14.575170 (+ 1250us) server_negotiation.cc:407] Connection header received
0619 04:38:14.576985 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:14.576990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:14.577077 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:14.577183 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:14.578246 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.579034 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.580624 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.581254 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.581388 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:14.582578 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:14.582583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:14.582586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:14.582604 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:14.582634 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:14.582646 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:14.582723 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:14.582911 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 04:38:14.583005 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 04:38:14.584223 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:14.574142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38813 (local address 127.0.97.3:33589)
0619 04:38:14.575005 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:14.575040 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:14.576715 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:14.577406 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:14.577414 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:14.577504 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:14.578058 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:14.578074 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.579173 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.579176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:14.580476 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:14.580492 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.581453 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.581458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:14.581664 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:14.582289 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:38:14.582292 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:14.582329 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:38:14.582408 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:14.582743 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:14.582752 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:14.582758 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 04:38:14.582893 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 04:38:14.583190 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":255,"threads_started":1}
I20260619 04:38:14.585709 20712 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38813
I20260619 04:38:14.586030 20712 heartbeater.cc:464] Registering TS with master...
I20260619 04:38:14.586800 20712 heartbeater.cc:511] Master 127.0.97.62:38813 requested a full tablet report, sending...
I20260619 04:38:14.588141 20265 ts_manager.cc:194] Registered new tserver with Master: 39444f08a2534d6ea56faa6c0e524760 (127.0.97.3:36623)
I20260619 04:38:14.588950 20265 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:33589
I20260619 04:38:14.595778 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:38:14.605218 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:14.599728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50526 (local address 127.0.97.62:38813)
0619 04:38:14.599912 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:14.599916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:14.600274 (+ 358us) server_negotiation.cc:407] Connection header received
0619 04:38:14.600401 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:14.600404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:14.600500 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:14.600582 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:14.600997 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.601665 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.602988 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.603379 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.603459 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:14.604436 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:14.604442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:14.604447 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:14.604489 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:14.604527 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:14.604543 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:14.604631 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:14.604787 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 04:38:14.604892 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:38:14.609299 20265 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:38:14.617266 20265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:38:14.653375 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:14.628746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36623 (local address 127.0.0.1:42186)
0619 04:38:14.644034 (+ 15288us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:14.644049 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:14.644148 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:14.645164 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:14.645171 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:14.645201 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:14.645630 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:14.645642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.647659 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.647664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:14.650418 (+ 2754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:14.650429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.652497 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.652504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:14.652704 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:14.652738 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 04:38:14.652823 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 04:38:14.653015 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15184,"mutex_wait_us":682,"thread_start_us":14156,"threads_started":1}
I20260619 04:38:14.653456 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:14.628795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.0.97.1:35601)
0619 04:38:14.643684 (+ 14889us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:14.643694 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:14.644262 (+ 568us) server_negotiation.cc:407] Connection header received
0619 04:38:14.644370 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:14.644378 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:14.645030 (+ 652us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:14.645203 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:14.646049 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.647515 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.650439 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.651845 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.652038 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:14.652302 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:14.652856 (+ 554us) server_negotiation.cc:299] Negotiation successful
0619 04:38:14.653104 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14753,"thread_start_us":14632,"threads_started":1}
I20260619 04:38:14.653474 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:14.629039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.0.97.3:36623)
0619 04:38:14.642505 (+ 13466us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:14.642516 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:14.644263 (+ 1747us) server_negotiation.cc:407] Connection header received
0619 04:38:14.644370 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:14.644378 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:14.644668 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:14.644859 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:14.645867 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.647512 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.650618 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.651935 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.652068 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:14.652342 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:14.652941 (+ 599us) server_negotiation.cc:299] Negotiation successful
0619 04:38:14.653118 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13234,"thread_start_us":13485,"threads_started":1}
I20260619 04:38:14.654042 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:14.628612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35601 (local address 127.0.0.1:48684)
0619 04:38:14.643785 (+ 15173us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:14.643834 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:14.644056 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:14.645439 (+ 1383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:14.645444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:14.645466 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:14.645860 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:14.645871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.647659 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.647664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:14.650171 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:14.650186 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.652432 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.652438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:14.652638 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:14.652671 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 04:38:14.652778 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 04:38:14.653789 (+ 1011us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14436,"mutex_wait_us":842,"thread_start_us":15148,"threads_started":1}
I20260619 04:38:14.658248 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:14.644193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41837 (local address 127.0.0.1:56130)
0619 04:38:14.648112 (+ 3919us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:14.648131 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:14.648238 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:14.651876 (+ 3638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:14.651881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:14.651978 (+ 97us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:14.652338 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:14.652351 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.654347 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.654352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:14.656143 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:14.656164 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.657515 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.657521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:14.657693 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:14.657726 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 04:38:14.657812 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 04:38:14.657942 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3783,"thread_start_us":3639,"threads_started":1}
I20260619 04:38:14.658313 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:14.644280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56130 (local address 127.0.97.2:41837)
0619 04:38:14.650967 (+ 6687us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:14.650976 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:14.651006 (+ 30us) server_negotiation.cc:407] Connection header received
0619 04:38:14.651097 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:14.651106 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:14.651370 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:14.652595 (+ 1225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:14.652834 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.654215 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.656327 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.657330 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.657450 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:14.657691 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:14.657863 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 04:38:14.658022 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6508,"thread_start_us":6385,"threads_started":1}
I20260619 04:38:14.658025 20386 tablet_service.cc:1467] Processing CreateTablet for tablet aaee7c08a63447fab7ee755a6e4ec0d0 (DEFAULT_TABLE table=test-table [id=33d9f5e901664bd79f25f930d13d6600]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:38:14.658026 20647 tablet_service.cc:1467] Processing CreateTablet for tablet aaee7c08a63447fab7ee755a6e4ec0d0 (DEFAULT_TABLE table=test-table [id=33d9f5e901664bd79f25f930d13d6600]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:38:14.659518 20386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaee7c08a63447fab7ee755a6e4ec0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:14.659516 20647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaee7c08a63447fab7ee755a6e4ec0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:14.661492 20517 tablet_service.cc:1467] Processing CreateTablet for tablet aaee7c08a63447fab7ee755a6e4ec0d0 (DEFAULT_TABLE table=test-table [id=33d9f5e901664bd79f25f930d13d6600]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:38:14.662544 20517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaee7c08a63447fab7ee755a6e4ec0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:14.673564 20730 tablet_bootstrap.cc:492] T aaee7c08a63447fab7ee755a6e4ec0d0 P 4f855a4f98204b348eb3f067527aadcb: Bootstrap starting.
I20260619 04:38:14.676807 20732 tablet_bootstrap.cc:492] T aaee7c08a63447fab7ee755a6e4ec0d0 P 39444f08a2534d6ea56faa6c0e524760: Bootstrap starting.
I20260619 04:38:14.677168 20730 tablet_bootstrap.cc:654] T aaee7c08a63447fab7ee755a6e4ec0d0 P 4f855a4f98204b348eb3f067527aadcb: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:14.678103 20730 log.cc:824] T aaee7c08a63447fab7ee755a6e4ec0d0 P 4f855a4f98204b348eb3f067527aadcb: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:14.680671 20732 tablet_bootstrap.cc:654] T aaee7c08a63447fab7ee755a6e4ec0d0 P 39444f08a2534d6ea56faa6c0e524760: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:14.681264 20730 tablet_bootstrap.cc:492] T aaee7c08a63447fab7ee755a6e4ec0d0 P 4f855a4f98204b348eb3f067527aadcb: No bootstrap required, opened a new log
I20260619 04:38:14.681499 20730 ts_tablet_manager.cc:1397] T aaee7c08a63447fab7ee755a6e4ec0d0 P 4f855a4f98204b348eb3f067527aadcb: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 04:38:14.681892 20732 log.cc:824] T aaee7c08a63447fab7ee755a6e4ec0d0 P 39444f08a2534d6ea56faa6c0e524760: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:14.685949 20731 tablet_bootstrap.cc:492] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0: Bootstrap starting.
I20260619 04:38:14.686147 20732 tablet_bootstrap.cc:492] T aaee7c08a63447fab7ee755a6e4ec0d0 P 39444f08a2534d6ea56faa6c0e524760: No bootstrap required, opened a new log
I20260619 04:38:14.686477 20732 ts_tablet_manager.cc:1397] T aaee7c08a63447fab7ee755a6e4ec0d0 P 39444f08a2534d6ea56faa6c0e524760: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260619 04:38:14.689555 20731 tablet_bootstrap.cc:654] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:14.690699 20731 log.cc:824] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:14.690693 20730 raft_consensus.cc:348] T aaee7c08a63447fab7ee755a6e4ec0d0 P 4f855a4f98204b348eb3f067527aadcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b79b3ec3f694980a4ced07c8c1f76d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35601 } } peers { permanent_uuid: "39444f08a2534d6ea56faa6c0e524760" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36623 } } peers { permanent_uuid: "4f855a4f98204b348eb3f067527aadcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41837 } }
I20260619 04:38:14.691115 20730 raft_consensus.cc:374] T aaee7c08a63447fab7ee755a6e4ec0d0 P 4f855a4f98204b348eb3f067527aadcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:14.691325 20730 raft_consensus.cc:729] T aaee7c08a63447fab7ee755a6e4ec0d0 P 4f855a4f98204b348eb3f067527aadcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f855a4f98204b348eb3f067527aadcb, State: Initialized, Role: FOLLOWER
I20260619 04:38:14.692070 20730 consensus_queue.cc:260] T aaee7c08a63447fab7ee755a6e4ec0d0 P 4f855a4f98204b348eb3f067527aadcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b79b3ec3f694980a4ced07c8c1f76d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35601 } } peers { permanent_uuid: "39444f08a2534d6ea56faa6c0e524760" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36623 } } peers { permanent_uuid: "4f855a4f98204b348eb3f067527aadcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41837 } }
I20260619 04:38:14.693830 20731 tablet_bootstrap.cc:492] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0: No bootstrap required, opened a new log
I20260619 04:38:14.694118 20731 ts_tablet_manager.cc:1397] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260619 04:38:14.694787 20732 raft_consensus.cc:348] T aaee7c08a63447fab7ee755a6e4ec0d0 P 39444f08a2534d6ea56faa6c0e524760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b79b3ec3f694980a4ced07c8c1f76d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35601 } } peers { permanent_uuid: "39444f08a2534d6ea56faa6c0e524760" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36623 } } peers { permanent_uuid: "4f855a4f98204b348eb3f067527aadcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41837 } }
I20260619 04:38:14.695104 20732 raft_consensus.cc:374] T aaee7c08a63447fab7ee755a6e4ec0d0 P 39444f08a2534d6ea56faa6c0e524760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:14.695188 20732 raft_consensus.cc:729] T aaee7c08a63447fab7ee755a6e4ec0d0 P 39444f08a2534d6ea56faa6c0e524760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39444f08a2534d6ea56faa6c0e524760, State: Initialized, Role: FOLLOWER
I20260619 04:38:14.695699 20732 consensus_queue.cc:260] T aaee7c08a63447fab7ee755a6e4ec0d0 P 39444f08a2534d6ea56faa6c0e524760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b79b3ec3f694980a4ced07c8c1f76d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35601 } } peers { permanent_uuid: "39444f08a2534d6ea56faa6c0e524760" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36623 } } peers { permanent_uuid: "4f855a4f98204b348eb3f067527aadcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41837 } }
I20260619 04:38:14.696672 20712 heartbeater.cc:503] Master 127.0.97.62:38813 was elected leader, sending a full tablet report...
I20260619 04:38:14.698845 20732 ts_tablet_manager.cc:1428] T aaee7c08a63447fab7ee755a6e4ec0d0 P 39444f08a2534d6ea56faa6c0e524760: Time spent starting tablet: real 0.012s user 0.008s sys 0.001s
I20260619 04:38:14.700098 20730 ts_tablet_manager.cc:1428] T aaee7c08a63447fab7ee755a6e4ec0d0 P 4f855a4f98204b348eb3f067527aadcb: Time spent starting tablet: real 0.019s user 0.013s sys 0.004s
I20260619 04:38:14.700583 20582 heartbeater.cc:503] Master 127.0.97.62:38813 was elected leader, sending a full tablet report...
I20260619 04:38:14.705511 20731 raft_consensus.cc:348] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b79b3ec3f694980a4ced07c8c1f76d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35601 } } peers { permanent_uuid: "39444f08a2534d6ea56faa6c0e524760" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36623 } } peers { permanent_uuid: "4f855a4f98204b348eb3f067527aadcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41837 } }
I20260619 04:38:14.705870 20731 raft_consensus.cc:374] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:14.706030 20731 raft_consensus.cc:729] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b79b3ec3f694980a4ced07c8c1f76d0, State: Initialized, Role: FOLLOWER
I20260619 04:38:14.706662 20731 consensus_queue.cc:260] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b79b3ec3f694980a4ced07c8c1f76d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35601 } } peers { permanent_uuid: "39444f08a2534d6ea56faa6c0e524760" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36623 } } peers { permanent_uuid: "4f855a4f98204b348eb3f067527aadcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41837 } }
I20260619 04:38:14.707674 20451 heartbeater.cc:503] Master 127.0.97.62:38813 was elected leader, sending a full tablet report...
I20260619 04:38:14.708150 20731 ts_tablet_manager.cc:1428] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
W20260619 04:38:14.822882 20713 tablet.cc:2369] T aaee7c08a63447fab7ee755a6e4ec0d0 P 39444f08a2534d6ea56faa6c0e524760: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:38:14.825239 20738 raft_consensus.cc:482] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:38:14.825525 20738 raft_consensus.cc:504] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b79b3ec3f694980a4ced07c8c1f76d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35601 } } peers { permanent_uuid: "39444f08a2534d6ea56faa6c0e524760" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36623 } } peers { permanent_uuid: "4f855a4f98204b348eb3f067527aadcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41837 } }
I20260619 04:38:14.827204 20738 leader_election.cc:288] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39444f08a2534d6ea56faa6c0e524760 (127.0.97.3:36623), 4f855a4f98204b348eb3f067527aadcb (127.0.97.2:41837)
I20260619 04:38:14.833669 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:14.827565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41325 (local address 127.0.97.3:36623)
0619 04:38:14.827785 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:14.827791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:14.828086 (+ 295us) server_negotiation.cc:407] Connection header received
0619 04:38:14.828281 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:14.828287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:14.828371 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:14.828480 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:14.829086 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.829975 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.831938 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.832777 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.832880 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:14.832950 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:14.833316 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 04:38:14.833436 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 04:38:14.833750 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:14.827434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36623 (local address 127.0.97.1:41325)
0619 04:38:14.827968 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:14.827988 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:14.828143 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:14.828527 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:14.828532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:14.828567 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:14.828934 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:14.828948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.830099 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.830105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:14.831735 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:14.831747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.832953 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.832959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:14.833157 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:14.833190 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 04:38:14.833264 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 04:38:14.833362 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":22,"thread_start_us":277,"threads_started":1}
I20260619 04:38:14.833771 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:14.828309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56515 (local address 127.0.97.2:41837)
0619 04:38:14.828493 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:14.828499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:14.828775 (+ 276us) server_negotiation.cc:407] Connection header received
0619 04:38:14.828911 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:14.828916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:14.829004 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:14.829085 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:38:14.829623 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.830409 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.832118 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.832945 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.833065 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:14.833148 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:14.833439 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 04:38:14.833558 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 04:38:14.834247 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:14.828096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41837 (local address 127.0.97.1:56515)
0619 04:38:14.828682 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:14.828695 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:14.828792 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:14.829111 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:14.829115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:14.829138 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:38:14.829496 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:14.829508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.830539 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.830543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:14.831944 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:14.831958 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.833113 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.833118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:14.833274 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:14.833308 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 04:38:14.833398 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 04:38:14.833506 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":18,"thread_start_us":397,"threads_started":1}
I20260619 04:38:14.834731 20667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaee7c08a63447fab7ee755a6e4ec0d0" candidate_uuid: "6b79b3ec3f694980a4ced07c8c1f76d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39444f08a2534d6ea56faa6c0e524760" is_pre_election: true
I20260619 04:38:14.835086 20537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaee7c08a63447fab7ee755a6e4ec0d0" candidate_uuid: "6b79b3ec3f694980a4ced07c8c1f76d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f855a4f98204b348eb3f067527aadcb" is_pre_election: true
I20260619 04:38:14.835192 20667 raft_consensus.cc:2454] T aaee7c08a63447fab7ee755a6e4ec0d0 P 39444f08a2534d6ea56faa6c0e524760 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b79b3ec3f694980a4ced07c8c1f76d0 in term 0.
I20260619 04:38:14.835506 20537 raft_consensus.cc:2454] T aaee7c08a63447fab7ee755a6e4ec0d0 P 4f855a4f98204b348eb3f067527aadcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b79b3ec3f694980a4ced07c8c1f76d0 in term 0.
I20260619 04:38:14.835847 20339 leader_election.cc:302] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39444f08a2534d6ea56faa6c0e524760, 6b79b3ec3f694980a4ced07c8c1f76d0; no voters:
I20260619 04:38:14.836180 20738 raft_consensus.cc:2781] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:38:14.836326 20738 raft_consensus.cc:482] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:38:14.836431 20738 raft_consensus.cc:3037] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:14.838615 20738 raft_consensus.cc:504] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b79b3ec3f694980a4ced07c8c1f76d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35601 } } peers { permanent_uuid: "39444f08a2534d6ea56faa6c0e524760" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36623 } } peers { permanent_uuid: "4f855a4f98204b348eb3f067527aadcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41837 } }
I20260619 04:38:14.839226 20738 leader_election.cc:288] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0 [CANDIDATE]: Term 1 election: Requested vote from peers 39444f08a2534d6ea56faa6c0e524760 (127.0.97.3:36623), 4f855a4f98204b348eb3f067527aadcb (127.0.97.2:41837)
I20260619 04:38:14.839665 20667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaee7c08a63447fab7ee755a6e4ec0d0" candidate_uuid: "6b79b3ec3f694980a4ced07c8c1f76d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39444f08a2534d6ea56faa6c0e524760"
I20260619 04:38:14.839742 20537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaee7c08a63447fab7ee755a6e4ec0d0" candidate_uuid: "6b79b3ec3f694980a4ced07c8c1f76d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f855a4f98204b348eb3f067527aadcb"
I20260619 04:38:14.839846 20667 raft_consensus.cc:3037] T aaee7c08a63447fab7ee755a6e4ec0d0 P 39444f08a2534d6ea56faa6c0e524760 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:14.839926 20537 raft_consensus.cc:3037] T aaee7c08a63447fab7ee755a6e4ec0d0 P 4f855a4f98204b348eb3f067527aadcb [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:14.842046 20667 raft_consensus.cc:2454] T aaee7c08a63447fab7ee755a6e4ec0d0 P 39444f08a2534d6ea56faa6c0e524760 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b79b3ec3f694980a4ced07c8c1f76d0 in term 1.
I20260619 04:38:14.842111 20537 raft_consensus.cc:2454] T aaee7c08a63447fab7ee755a6e4ec0d0 P 4f855a4f98204b348eb3f067527aadcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b79b3ec3f694980a4ced07c8c1f76d0 in term 1.
I20260619 04:38:14.842432 20339 leader_election.cc:302] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39444f08a2534d6ea56faa6c0e524760, 6b79b3ec3f694980a4ced07c8c1f76d0; no voters:
I20260619 04:38:14.842690 20738 raft_consensus.cc:2781] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:14.843099 20738 raft_consensus.cc:686] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0 [term 1 LEADER]: Becoming Leader. State: Replica: 6b79b3ec3f694980a4ced07c8c1f76d0, State: Running, Role: LEADER
I20260619 04:38:14.843539 20738 consensus_queue.cc:237] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b79b3ec3f694980a4ced07c8c1f76d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35601 } } peers { permanent_uuid: "39444f08a2534d6ea56faa6c0e524760" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36623 } } peers { permanent_uuid: "4f855a4f98204b348eb3f067527aadcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41837 } }
I20260619 04:38:14.847915 20264 catalog_manager.cc:5495] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b79b3ec3f694980a4ced07c8c1f76d0 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "6b79b3ec3f694980a4ced07c8c1f76d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b79b3ec3f694980a4ced07c8c1f76d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39444f08a2534d6ea56faa6c0e524760" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f855a4f98204b348eb3f067527aadcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41837 } health_report { overall_health: UNKNOWN } } }
W20260619 04:38:14.888607 20583 tablet.cc:2369] T aaee7c08a63447fab7ee755a6e4ec0d0 P 4f855a4f98204b348eb3f067527aadcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:38:14.902281 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:14.896927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48696 (local address 127.0.97.1:35601)
0619 04:38:14.897095 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:14.897100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:14.897217 (+ 117us) server_negotiation.cc:407] Connection header received
0619 04:38:14.897423 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:14.897428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:14.897523 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:14.897640 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:14.898097 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.898869 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.900301 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.900763 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.900864 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:14.901290 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:14.901787 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:14.901898 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:14.901940 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 04:38:14.902010 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 04:38:14.910131 20537 raft_consensus.cc:1261] T aaee7c08a63447fab7ee755a6e4ec0d0 P 4f855a4f98204b348eb3f067527aadcb [term 1 FOLLOWER]: Refusing update from remote peer 6b79b3ec3f694980a4ced07c8c1f76d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:38:14.910198 20667 raft_consensus.cc:1261] T aaee7c08a63447fab7ee755a6e4ec0d0 P 39444f08a2534d6ea56faa6c0e524760 [term 1 FOLLOWER]: Refusing update from remote peer 6b79b3ec3f694980a4ced07c8c1f76d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:38:14.911002 20741 consensus_queue.cc:1035] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f855a4f98204b348eb3f067527aadcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:38:14.911387 20738 consensus_queue.cc:1035] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39444f08a2534d6ea56faa6c0e524760" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:38:14.922752 20744 mvcc.cc:205] Tried to move back new op lower bound from 7298432593542402048 to 7298432593283276800. Current Snapshot: MvccSnapshot[applied={T|T < 7298432593542402048}]
I20260619 04:38:14.923645 20746 mvcc.cc:205] Tried to move back new op lower bound from 7298432593542402048 to 7298432593283276800. Current Snapshot: MvccSnapshot[applied={T|T < 7298432593542402048}]
I20260619 04:38:14.928673 20745 mvcc.cc:205] Tried to move back new op lower bound from 7298432593542402048 to 7298432593283276800. Current Snapshot: MvccSnapshot[applied={T|T < 7298432593542402048}]
I20260619 04:38:14.937724 20264 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:38:14.937968 20264 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:38:14.941362 20264 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c65d6b1c2d44b3faff72c97d6254c34: Sending DeleteTablet for 3 replicas of tablet aaee7c08a63447fab7ee755a6e4ec0d0
I20260619 04:38:14.942425 20386 tablet_service.cc:1514] Processing DeleteTablet for tablet aaee7c08a63447fab7ee755a6e4ec0d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:38:14 UTC) from {username='slave'} at 127.0.0.1:48684
I20260619 04:38:14.942581 20517 tablet_service.cc:1514] Processing DeleteTablet for tablet aaee7c08a63447fab7ee755a6e4ec0d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:38:14 UTC) from {username='slave'} at 127.0.0.1:56130
I20260619 04:38:14.943218 20647 tablet_service.cc:1514] Processing DeleteTablet for tablet aaee7c08a63447fab7ee755a6e4ec0d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:38:14 UTC) from {username='slave'} at 127.0.0.1:42186
I20260619 04:38:14.943408 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 20324
I20260619 04:38:14.952555 20756 tablet_replica.cc:330] T aaee7c08a63447fab7ee755a6e4ec0d0 P 39444f08a2534d6ea56faa6c0e524760: stopping tablet replica
I20260619 04:38:14.953392 20755 tablet_replica.cc:330] T aaee7c08a63447fab7ee755a6e4ec0d0 P 4f855a4f98204b348eb3f067527aadcb: stopping tablet replica
I20260619 04:38:14.953617 20754 tablet_replica.cc:330] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0: stopping tablet replica
I20260619 04:38:14.953822 20755 raft_consensus.cc:2229] T aaee7c08a63447fab7ee755a6e4ec0d0 P 4f855a4f98204b348eb3f067527aadcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:38:14.953955 20754 raft_consensus.cc:2229] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:38:14.954326 20755 raft_consensus.cc:2258] T aaee7c08a63447fab7ee755a6e4ec0d0 P 4f855a4f98204b348eb3f067527aadcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:38:14.954588 20754 raft_consensus.cc:2258] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:38:14.955006 20756 raft_consensus.cc:2229] T aaee7c08a63447fab7ee755a6e4ec0d0 P 39444f08a2534d6ea56faa6c0e524760 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:38:14.955513 20756 raft_consensus.cc:2258] T aaee7c08a63447fab7ee755a6e4ec0d0 P 39444f08a2534d6ea56faa6c0e524760 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:38:14.957487 20756 ts_tablet_manager.cc:1905] T aaee7c08a63447fab7ee755a6e4ec0d0 P 39444f08a2534d6ea56faa6c0e524760: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:38:14.957496 20754 ts_tablet_manager.cc:1905] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:38:14.957489 20755 ts_tablet_manager.cc:1905] T aaee7c08a63447fab7ee755a6e4ec0d0 P 4f855a4f98204b348eb3f067527aadcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:38:14.964128 20755 ts_tablet_manager.cc:1918] T aaee7c08a63447fab7ee755a6e4ec0d0 P 4f855a4f98204b348eb3f067527aadcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:38:14.964290 20755 log.cc:1196] T aaee7c08a63447fab7ee755a6e4ec0d0 P 4f855a4f98204b348eb3f067527aadcb: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-1/wal/wals/aaee7c08a63447fab7ee755a6e4ec0d0
I20260619 04:38:14.964794 20755 ts_tablet_manager.cc:1939] T aaee7c08a63447fab7ee755a6e4ec0d0 P 4f855a4f98204b348eb3f067527aadcb: Deleting consensus metadata
I20260619 04:38:14.965466 20754 ts_tablet_manager.cc:1918] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:38:14.965610 20754 log.cc:1196] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-0/wal/wals/aaee7c08a63447fab7ee755a6e4ec0d0
I20260619 04:38:14.966110 20754 ts_tablet_manager.cc:1939] T aaee7c08a63447fab7ee755a6e4ec0d0 P 6b79b3ec3f694980a4ced07c8c1f76d0: Deleting consensus metadata
I20260619 04:38:14.966692 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:14.945962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48706 (local address 127.0.97.1:35601)
0619 04:38:14.946403 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:14.946410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:14.951374 (+ 4964us) server_negotiation.cc:407] Connection header received
0619 04:38:14.952292 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:14.952299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:14.952415 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:14.952515 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:14.953693 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.954888 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.961501 (+ 6613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:14.962155 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:14.962282 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:14.964804 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:14.964816 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:14.964834 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:14.965633 (+ 799us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:14.965709 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:14.965738 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:14.965827 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:14.966196 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 04:38:14.966304 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321}
I20260619 04:38:14.966696 20249 catalog_manager.cc:4841] TS 4f855a4f98204b348eb3f067527aadcb (127.0.97.2:41837): tablet aaee7c08a63447fab7ee755a6e4ec0d0 (table test-table [id=33d9f5e901664bd79f25f930d13d6600]) successfully deleted
I20260619 04:38:14.968343 20446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:14.968487 20250 catalog_manager.cc:4841] TS 6b79b3ec3f694980a4ced07c8c1f76d0 (127.0.97.1:35601): tablet aaee7c08a63447fab7ee755a6e4ec0d0 (table test-table [id=33d9f5e901664bd79f25f930d13d6600]) successfully deleted
I20260619 04:38:14.970320 20756 ts_tablet_manager.cc:1918] T aaee7c08a63447fab7ee755a6e4ec0d0 P 39444f08a2534d6ea56faa6c0e524760: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:38:14.970490 20756 log.cc:1196] T aaee7c08a63447fab7ee755a6e4ec0d0 P 39444f08a2534d6ea56faa6c0e524760: Deleting WAL directory at /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781843588881932-388-0/minicluster-data/ts-2/wal/wals/aaee7c08a63447fab7ee755a6e4ec0d0
I20260619 04:38:14.970973 20756 ts_tablet_manager.cc:1939] T aaee7c08a63447fab7ee755a6e4ec0d0 P 39444f08a2534d6ea56faa6c0e524760: Deleting consensus metadata
I20260619 04:38:14.972563 20249 catalog_manager.cc:4841] TS 39444f08a2534d6ea56faa6c0e524760 (127.0.97.3:36623): tablet aaee7c08a63447fab7ee755a6e4ec0d0 (table test-table [id=33d9f5e901664bd79f25f930d13d6600]) successfully deleted
I20260619 04:38:15.061450 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 20324
I20260619 04:38:15.075883 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 20455
I20260619 04:38:15.082723 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:15.076630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56142 (local address 127.0.97.2:41837)
0619 04:38:15.076801 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:15.076807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:15.076827 (+ 20us) server_negotiation.cc:407] Connection header received
0619 04:38:15.077010 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:15.077016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:15.077121 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:15.077205 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:15.077609 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:15.078331 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:15.079583 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:15.080003 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:15.080108 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:15.081285 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:15.081294 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:15.081344 (+ 50us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:15.081910 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:15.081962 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:15.081982 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:15.082172 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:15.082294 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 04:38:15.082413 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 04:38:15.083372 20577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:15.171027 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 20455
I20260619 04:38:15.184623 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 20585
I20260619 04:38:15.198362 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:15.185379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.0.97.3:36623)
0619 04:38:15.185556 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:15.185561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:15.185578 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:38:15.185641 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:15.185647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:15.185762 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:15.185857 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:15.186352 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:15.187128 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:15.188430 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:15.188844 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:15.188957 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:15.196971 (+ 8014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:15.196983 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:15.197002 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:15.197604 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:15.197657 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:15.197680 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:15.197795 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:15.197941 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 04:38:15.198059 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:38:15.199038 20707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:15.285327 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 20585
I20260619 04:38:15.299013 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 20234
I20260619 04:38:15.300020 20295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:15.407570 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 20234
2026-06-19T04:38:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2730 ms)
[----------] 6 tests from EncryptionPolicyTest (20554 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:38:15 dist-test-slave-n3tq krb5kdc[20764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:38:15 dist-test-slave-n3tq krb5kdc[20764](info): set up 2 sockets
Jun 19 04:38:15 dist-test-slave-n3tq krb5kdc[20764](info): commencing operation
krb5kdc: starting...
W20260619 04:38:16.416546 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:38:16 dist-test-slave-n3tq krb5kdc[20764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:38:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:38:16.661731 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:16.974195 20777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:16.974489 20777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:16.974615 20777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:16.981758 20777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:16.981875 20777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:16.981951 20777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:16.982033 20777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:16.982088 20777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:16.991753 20777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38885
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:16.993036 20777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:16.994277 20777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:17.005163 20785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:17.005119 20783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:17.005118 20782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:17.006039 20777 server_base.cc:1034] running on GCE node
I20260619 04:38:17.006722 20777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:17.007978 20777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:17.009172 20777 hybrid_clock.cc:648] HybridClock initialized: now 1781843897009135 us; error 27 us; skew 500 ppm
I20260619 04:38:17.009497 20777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:17 dist-test-slave-n3tq krb5kdc[20764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:17.012413 20777 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:38:17.013772 20777 webserver.cc:460] Webserver started at http://127.0.0.1:32819/ using document root <none> and password file <none>
I20260619 04:38:17.014375 20777 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:17.014467 20777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:17.014701 20777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:17.017518 20777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "3abdd00b4047486bba795870b4cfb713"
format_stamp: "Formatted at 2026-06-19 04:38:17 on dist-test-slave-n3tq"
server_key: "4a00b1da0b6f8747f448753018caeefd"
server_key_iv: "1d4f60dd78e685c50394e1159753fde1"
server_key_version: "encryptionkey@0"
I20260619 04:38:17.018210 20777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "3abdd00b4047486bba795870b4cfb713"
format_stamp: "Formatted at 2026-06-19 04:38:17 on dist-test-slave-n3tq"
server_key: "4a00b1da0b6f8747f448753018caeefd"
server_key_iv: "1d4f60dd78e685c50394e1159753fde1"
server_key_version: "encryptionkey@0"
I20260619 04:38:17.022921 20777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 04:38:17.026083 20792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:17.027338 20777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:38:17.027498 20777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "3abdd00b4047486bba795870b4cfb713"
format_stamp: "Formatted at 2026-06-19 04:38:17 on dist-test-slave-n3tq"
server_key: "4a00b1da0b6f8747f448753018caeefd"
server_key_iv: "1d4f60dd78e685c50394e1159753fde1"
server_key_version: "encryptionkey@0"
I20260619 04:38:17.027665 20777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:17.041289 20777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:17.045295 20777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:17.045537 20777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:17.062259 20777 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38885
I20260619 04:38:17.062282 20843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38885 every 8 connection(s)
I20260619 04:38:17.063611 20777 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:17.067238 20844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:17.073537 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 20777
I20260619 04:38:17.073761 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:17.074548 388 external_mini_cluster.cc:1428] Setting key 602a9bf02145ad6dde625f1a32e0c4d7
I20260619 04:38:17.077911 20844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713: Bootstrap starting.
I20260619 04:38:17.081920 20844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:17.083153 20844 log.cc:824] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:17.085996 20844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713: No bootstrap required, opened a new log
Jun 19 04:38:17 dist-test-slave-n3tq krb5kdc[20764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 04:38:17.094937 20844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abdd00b4047486bba795870b4cfb713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } }
I20260619 04:38:17.095392 20844 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:17.095484 20844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3abdd00b4047486bba795870b4cfb713, State: Initialized, Role: FOLLOWER
I20260619 04:38:17.096177 20844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abdd00b4047486bba795870b4cfb713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } }
I20260619 04:38:17.096443 20844 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:38:17.096629 20844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:38:17.096812 20844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:17.099638 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:17.076635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59933 (local address 127.0.0.1:38885)
0619 04:38:17.077573 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:17.077591 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:17.077965 (+ 374us) server_negotiation.cc:407] Connection header received
0619 04:38:17.080470 (+ 2505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:17.080499 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:17.081020 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:17.081470 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:17.082815 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:17.084501 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:17.085799 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:17.086805 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:17.086980 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:17.090868 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:17.090879 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:17.090901 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:17.090951 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:17.094842 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:17.095717 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:17.095726 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:17.095734 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:17.095941 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:17.096354 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:17.096358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:17.096361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:17.097153 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:17.097471 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:17.097946 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 04:38:17.098292 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":420,"threads_started":1}
I20260619 04:38:17.100009 20844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abdd00b4047486bba795870b4cfb713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } }
I20260619 04:38:17.100647 20844 leader_election.cc:302] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3abdd00b4047486bba795870b4cfb713; no voters:
I20260619 04:38:17.101300 20844 leader_election.cc:288] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:38:17.101435 20849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:17.102916 20849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713 [term 1 LEADER]: Becoming Leader. State: Replica: 3abdd00b4047486bba795870b4cfb713, State: Running, Role: LEADER
I20260619 04:38:17.103617 20844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:17.103482 20849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abdd00b4047486bba795870b4cfb713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } }
I20260619 04:38:17.104226 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:38:17.109122 20850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3abdd00b4047486bba795870b4cfb713" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abdd00b4047486bba795870b4cfb713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } } }
I20260619 04:38:17.109467 20850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:17.109959 20851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3abdd00b4047486bba795870b4cfb713. Latest consensus state: current_term: 1 leader_uuid: "3abdd00b4047486bba795870b4cfb713" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abdd00b4047486bba795870b4cfb713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } } }
I20260619 04:38:17.110249 20851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:17.114777 20856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:17.120730 20856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:17.142227 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:17.117926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57037 (local address 127.0.0.1:38885)
0619 04:38:17.118091 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:17.118097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:17.125524 (+ 7427us) server_negotiation.cc:407] Connection header received
0619 04:38:17.125737 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:17.125744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:17.125827 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:17.125899 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:17.127941 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:17.129133 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:17.130926 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:17.131586 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:17.131694 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:17.134966 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:17.134972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:17.134977 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:17.135016 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:17.138581 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:17.139538 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:17.139544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:17.139547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:17.139708 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:17.140282 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:17.140288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:17.140291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:17.140749 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:17.140977 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:17.141689 (+ 712us) server_negotiation.cc:299] Negotiation successful
0619 04:38:17.141876 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 04:38:17.143550 20856 catalog_manager.cc:1347] Generated new cluster ID: 968ab779fcb34c49b6bb369c3c0b5b33
I20260619 04:38:17.143673 20856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:17.167280 20856 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:17.168618 20856 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:17.180148 20856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3abdd00b4047486bba795870b4cfb713: Generated new TSK 0
I20260619 04:38:17.181007 20856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:38:17.194564 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 20777
I20260619 04:38:17.195695 20839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:17.288749 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 20777
W20260619 04:38:17.303113 20762 connection.cc:535] client connection to 127.0.0.1:38885 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38885): Connection reset by peer (error 104)
2026-06-19T04:38:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1880 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:38:17 dist-test-slave-n3tq krb5kdc[20877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:38:17 dist-test-slave-n3tq krb5kdc[20877](info): set up 2 sockets
Jun 19 04:38:17 dist-test-slave-n3tq krb5kdc[20877](info): commencing operation
krb5kdc: starting...
W20260619 04:38:18.300760 388 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:38:18 dist-test-slave-n3tq krb5kdc[20877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:38:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:38:18.527160 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42033
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:18.848799 20890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:18.849078 20890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:18.849205 20890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:18.856354 20890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:18.856492 20890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:18.856566 20890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:18.856649 20890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:18.856706 20890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:18.866518 20890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42033
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:18.867823 20890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:18.869174 20890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:18.880178 20895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:18.880123 20896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:18.880198 20898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:18.880863 20890 server_base.cc:1034] running on GCE node
I20260619 04:38:18.881572 20890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:18.882932 20890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:18.884135 20890 hybrid_clock.cc:648] HybridClock initialized: now 1781843898884081 us; error 43 us; skew 500 ppm
I20260619 04:38:18.884488 20890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:18 dist-test-slave-n3tq krb5kdc[20877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:18.887588 20890 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:38:18.889058 20890 webserver.cc:460] Webserver started at http://127.0.0.1:44407/ using document root <none> and password file <none>
I20260619 04:38:18.889716 20890 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:18.889827 20890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:18.890105 20890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:18.893035 20890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "7b264cc27e724e5d8fe0c0c7e5a14990"
format_stamp: "Formatted at 2026-06-19 04:38:18 on dist-test-slave-n3tq"
server_key: "6ec43790344ce08ec67c5b844fb48b82"
server_key_iv: "e20b656449909d35122a96cc3635b520"
server_key_version: "encryptionkey@0"
I20260619 04:38:18.893767 20890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "7b264cc27e724e5d8fe0c0c7e5a14990"
format_stamp: "Formatted at 2026-06-19 04:38:18 on dist-test-slave-n3tq"
server_key: "6ec43790344ce08ec67c5b844fb48b82"
server_key_iv: "e20b656449909d35122a96cc3635b520"
server_key_version: "encryptionkey@0"
I20260619 04:38:18.898828 20890 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 04:38:18.902117 20905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:18.903348 20890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:38:18.903492 20890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "7b264cc27e724e5d8fe0c0c7e5a14990"
format_stamp: "Formatted at 2026-06-19 04:38:18 on dist-test-slave-n3tq"
server_key: "6ec43790344ce08ec67c5b844fb48b82"
server_key_iv: "e20b656449909d35122a96cc3635b520"
server_key_version: "encryptionkey@0"
I20260619 04:38:18.903672 20890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:18.922030 20890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:18.926314 20890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:18.926573 20890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:18.943235 20890 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42033
I20260619 04:38:18.943265 20956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42033 every 8 connection(s)
I20260619 04:38:18.944564 20890 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:18.948079 20957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:18.948764 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 20890
I20260619 04:38:18.948927 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:18.949322 388 external_mini_cluster.cc:1428] Setting key 44ee1dba1e66caa4ec5671ae659ea1a8
I20260619 04:38:18.961825 20957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990: Bootstrap starting.
I20260619 04:38:18.963639 20960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:18.951009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53499 (local address 127.0.0.1:42033)
0619 04:38:18.951758 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:18.951773 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:18.951822 (+ 49us) server_negotiation.cc:407] Connection header received
0619 04:38:18.954184 (+ 2362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:18.954218 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:18.954725 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:18.955158 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:18.955650 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:18.957493 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:18.958804 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:18.959838 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:18.959994 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:18.961035 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:18.961046 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:18.961063 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:18.961739 (+ 676us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:18.961834 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:18.961863 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:18.962006 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:18.962206 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 04:38:18.962380 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":302,"threads_started":1}
I20260619 04:38:18.965437 20957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:18.966534 20957 log.cc:824] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:18.967576 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:38:18.969161 20957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990: No bootstrap required, opened a new log
I20260619 04:38:18.978068 20960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:18.971672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35183 (local address 127.0.0.1:42033)
0619 04:38:18.971860 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:18.971867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:18.972156 (+ 289us) server_negotiation.cc:407] Connection header received
0619 04:38:18.972290 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:18.972295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:18.972370 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:18.972488 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:18.972975 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:18.974110 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:18.975432 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:18.976129 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:18.976257 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:18.977375 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:18.977381 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:18.977385 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:18.977406 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:18.977465 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:18.977487 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:18.977597 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:18.977718 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 04:38:18.977769 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 04:38:18.978070 20957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b264cc27e724e5d8fe0c0c7e5a14990" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42033 } }
I20260619 04:38:18.978549 20957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:18.978657 20957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b264cc27e724e5d8fe0c0c7e5a14990, State: Initialized, Role: FOLLOWER
I20260619 04:38:18.979290 20957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b264cc27e724e5d8fe0c0c7e5a14990" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42033 } }
I20260619 04:38:18.979527 20957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:38:18.979663 20957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:38:18.979809 20957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:18.981986 20957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b264cc27e724e5d8fe0c0c7e5a14990" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42033 } }
I20260619 04:38:18.982442 20957 leader_election.cc:302] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b264cc27e724e5d8fe0c0c7e5a14990; no voters:
I20260619 04:38:18.982975 20957 leader_election.cc:288] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:38:18.983104 20967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:18.984886 20957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:18.989279 20967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990 [term 1 LEADER]: Becoming Leader. State: Replica: 7b264cc27e724e5d8fe0c0c7e5a14990, State: Running, Role: LEADER
I20260619 04:38:18.989828 20967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b264cc27e724e5d8fe0c0c7e5a14990" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42033 } }
I20260619 04:38:18.992726 20968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b264cc27e724e5d8fe0c0c7e5a14990" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b264cc27e724e5d8fe0c0c7e5a14990" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42033 } } }
I20260619 04:38:18.992658 20969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b264cc27e724e5d8fe0c0c7e5a14990. Latest consensus state: current_term: 1 leader_uuid: "7b264cc27e724e5d8fe0c0c7e5a14990" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b264cc27e724e5d8fe0c0c7e5a14990" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42033 } } }
I20260619 04:38:18.993127 20968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:18.993227 20969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:18.994269 20982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:18.998775 20982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:19.007629 20982 catalog_manager.cc:1347] Generated new cluster ID: 540b1929c64a4291af92a6ee70db111f
I20260619 04:38:19.007753 20982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:19.026038 20982 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:19.027998 20982 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:19.039656 20982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b264cc27e724e5d8fe0c0c7e5a14990: Generated new TSK 0
I20260619 04:38:19.040547 20982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:38:19.053726 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 20890
W20260619 04:38:19.054909 20952 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53499
W20260619 04:38:19.055702 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 20890: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:38:19.055871 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 20890
W20260619 04:38:19.069650 20875 connection.cc:535] client connection to 127.0.0.1:42033 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42033): Connection reset by peer (error 104)
2026-06-19T04:38:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1762 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:38:19 dist-test-slave-n3tq krb5kdc[20988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:38:19 dist-test-slave-n3tq krb5kdc[20988](info): set up 2 sockets
Jun 19 04:38:19 dist-test-slave-n3tq krb5kdc[20988](info): commencing operation
krb5kdc: starting...
W20260619 04:38:20.068437 388 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:38:20 dist-test-slave-n3tq krb5kdc[20988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:38:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:38:20.297438 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39395
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:20.618791 21001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:20.619081 21001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:20.619213 21001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:20.626435 21001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:20.626554 21001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:20.626629 21001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:20.626706 21001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:20.626762 21001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:20.636653 21001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39395
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:20.637914 21001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:20.639184 21001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:20.650054 21009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:20.650002 21006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:20.650094 21007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:20.651141 21001 server_base.cc:1034] running on GCE node
I20260619 04:38:20.651849 21001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:20.653321 21001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:20.654510 21001 hybrid_clock.cc:648] HybridClock initialized: now 1781843900654461 us; error 45 us; skew 500 ppm
I20260619 04:38:20.654829 21001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:20 dist-test-slave-n3tq krb5kdc[20988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:20.660362 21001 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:38:20.661837 21001 webserver.cc:460] Webserver started at http://127.0.0.1:45183/ using document root <none> and password file <none>
I20260619 04:38:20.662492 21001 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:20.662601 21001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:20.662878 21001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:20.665807 21001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "c42caa7e76fa44bd9f57e0c30d0767cc"
format_stamp: "Formatted at 2026-06-19 04:38:20 on dist-test-slave-n3tq"
server_key: "342b2a0d952f6af8cece54ba8df34620"
server_key_iv: "9fc03a25b456f1b14c173c65b0cfc801"
server_key_version: "encryptionkey@0"
I20260619 04:38:20.666508 21001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "c42caa7e76fa44bd9f57e0c30d0767cc"
format_stamp: "Formatted at 2026-06-19 04:38:20 on dist-test-slave-n3tq"
server_key: "342b2a0d952f6af8cece54ba8df34620"
server_key_iv: "9fc03a25b456f1b14c173c65b0cfc801"
server_key_version: "encryptionkey@0"
I20260619 04:38:20.671556 21001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:38:20.674891 21016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:20.676182 21001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:38:20.676342 21001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "c42caa7e76fa44bd9f57e0c30d0767cc"
format_stamp: "Formatted at 2026-06-19 04:38:20 on dist-test-slave-n3tq"
server_key: "342b2a0d952f6af8cece54ba8df34620"
server_key_iv: "9fc03a25b456f1b14c173c65b0cfc801"
server_key_version: "encryptionkey@0"
I20260619 04:38:20.676573 21001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:20.682206 21001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:20.685962 21001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:20.686234 21001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:20.703207 21001 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39395
I20260619 04:38:20.703239 21067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39395 every 8 connection(s)
I20260619 04:38:20.704573 21001 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:20.708165 21068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:20.709362 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 21001
I20260619 04:38:20.709522 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:20.709924 388 external_mini_cluster.cc:1428] Setting key 1e010027bf0540d2e4e47e90a7d96c0a
I20260619 04:38:20.718055 21071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:20.711686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39225 (local address 127.0.0.1:39395)
0619 04:38:20.712480 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:20.712495 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:20.712558 (+ 63us) server_negotiation.cc:407] Connection header received
0619 04:38:20.714404 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:20.714432 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:20.714894 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:20.715331 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:20.716233 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:20.716241 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:20.716257 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:20.716272 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:20.716332 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:20.716361 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:20.716595 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:20.716714 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 04:38:20.716915 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":301,"threads_started":1}
I20260619 04:38:20.721360 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:38:20.721549 21068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc: Bootstrap starting.
I20260619 04:38:20.724422 21068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:20.725607 21068 log.cc:824] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:20.727504 21071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:20.725255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50035 (local address 127.0.0.1:39395)
0619 04:38:20.725407 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:20.725413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:20.725784 (+ 371us) server_negotiation.cc:407] Connection header received
0619 04:38:20.725952 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:20.725957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:20.726014 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:20.726123 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:20.726940 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:20.726944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:20.726948 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:20.726965 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:20.727006 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:20.727023 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:20.727129 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:20.727172 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 04:38:20.727247 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 04:38:20.728523 21068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc: No bootstrap required, opened a new log
I20260619 04:38:20.735421 21068 raft_consensus.cc:348] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42caa7e76fa44bd9f57e0c30d0767cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39395 } }
I20260619 04:38:20.735786 21068 raft_consensus.cc:374] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:20.735873 21068 raft_consensus.cc:729] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42caa7e76fa44bd9f57e0c30d0767cc, State: Initialized, Role: FOLLOWER
I20260619 04:38:20.736431 21068 consensus_queue.cc:260] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42caa7e76fa44bd9f57e0c30d0767cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39395 } }
I20260619 04:38:20.736704 21068 raft_consensus.cc:388] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:38:20.736832 21068 raft_consensus.cc:482] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:38:20.736999 21068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:20.739087 21068 raft_consensus.cc:504] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42caa7e76fa44bd9f57e0c30d0767cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39395 } }
I20260619 04:38:20.739562 21068 leader_election.cc:302] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c42caa7e76fa44bd9f57e0c30d0767cc; no voters:
I20260619 04:38:20.740096 21068 leader_election.cc:288] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:38:20.740209 21078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:20.740804 21078 raft_consensus.cc:686] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc [term 1 LEADER]: Becoming Leader. State: Replica: c42caa7e76fa44bd9f57e0c30d0767cc, State: Running, Role: LEADER
I20260619 04:38:20.741294 21078 consensus_queue.cc:237] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42caa7e76fa44bd9f57e0c30d0767cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39395 } }
I20260619 04:38:20.742013 21068 sys_catalog.cc:564] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:20.743607 21079 sys_catalog.cc:455] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c42caa7e76fa44bd9f57e0c30d0767cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42caa7e76fa44bd9f57e0c30d0767cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39395 } } }
I20260619 04:38:20.743906 21079 sys_catalog.cc:458] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:20.744699 21084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:20.744551 21080 sys_catalog.cc:455] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c42caa7e76fa44bd9f57e0c30d0767cc. Latest consensus state: current_term: 1 leader_uuid: "c42caa7e76fa44bd9f57e0c30d0767cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42caa7e76fa44bd9f57e0c30d0767cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39395 } } }
I20260619 04:38:20.745026 21080 sys_catalog.cc:458] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:20.753633 21084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:20.762352 21084 catalog_manager.cc:1347] Generated new cluster ID: 6577ad4a080c47f2a07717c7b596f4ec
I20260619 04:38:20.762454 21084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:20.777879 21084 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:20.778276 21084 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:20.786194 21084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c42caa7e76fa44bd9f57e0c30d0767cc: Generated new TSK 0
I20260619 04:38:20.787266 21084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:38:20.798347 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 21001
W20260619 04:38:20.799382 21063 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39225
W20260619 04:38:20.800014 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 21001: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:38:20.800153 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 21001
2026-06-19T04:38:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1743 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:38:20 dist-test-slave-n3tq krb5kdc[21099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:38:20 dist-test-slave-n3tq krb5kdc[21099](info): set up 2 sockets
Jun 19 04:38:20 dist-test-slave-n3tq krb5kdc[21099](info): commencing operation
krb5kdc: starting...
W20260619 04:38:21.808424 388 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:38:21 dist-test-slave-n3tq krb5kdc[21099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:38:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:38:22.040903 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33793
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:22.357080 21112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:22.357381 21112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:22.357519 21112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:22.364880 21112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:22.365002 21112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:22.365073 21112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:22.365157 21112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:22.365218 21112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:22.375288 21112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:22.376605 21112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:22.377902 21112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:22.388909 21118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:22.388948 21120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:22.388979 21117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:22.389932 21112 server_base.cc:1034] running on GCE node
I20260619 04:38:22.390647 21112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:22.392009 21112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:22.393230 21112 hybrid_clock.cc:648] HybridClock initialized: now 1781843902393165 us; error 49 us; skew 500 ppm
I20260619 04:38:22.393623 21112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:22 dist-test-slave-n3tq krb5kdc[21099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:22.396652 21112 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:38:22.398135 21112 webserver.cc:460] Webserver started at http://127.0.0.1:36831/ using document root <none> and password file <none>
I20260619 04:38:22.398789 21112 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:22.398903 21112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:22.399196 21112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:22.402122 21112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "831b56c27efc47e1ba19d122d02b3f31"
format_stamp: "Formatted at 2026-06-19 04:38:22 on dist-test-slave-n3tq"
server_key: "aa73d56d82384b5d6373d83779565ef8"
server_key_iv: "380b4a85d1778d9291acad520f7069ab"
server_key_version: "encryptionkey@0"
I20260619 04:38:22.402830 21112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "831b56c27efc47e1ba19d122d02b3f31"
format_stamp: "Formatted at 2026-06-19 04:38:22 on dist-test-slave-n3tq"
server_key: "aa73d56d82384b5d6373d83779565ef8"
server_key_iv: "380b4a85d1778d9291acad520f7069ab"
server_key_version: "encryptionkey@0"
I20260619 04:38:22.407703 21112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:38:22.410955 21127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:22.412171 21112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:38:22.412330 21112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "831b56c27efc47e1ba19d122d02b3f31"
format_stamp: "Formatted at 2026-06-19 04:38:22 on dist-test-slave-n3tq"
server_key: "aa73d56d82384b5d6373d83779565ef8"
server_key_iv: "380b4a85d1778d9291acad520f7069ab"
server_key_version: "encryptionkey@0"
I20260619 04:38:22.412535 21112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:22.448565 21112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:22.452994 21112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:22.453265 21112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:22.470695 21112 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33793
I20260619 04:38:22.470731 21178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33793 every 8 connection(s)
I20260619 04:38:22.472030 21112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:22.473281 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 21112
I20260619 04:38:22.473464 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:22.473966 388 external_mini_cluster.cc:1428] Setting key 8059ff47a81261774959f21d537c74d2
I20260619 04:38:22.476580 21179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 04:38:22 dist-test-slave-n3tq krb5kdc[21099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 04:38:22.489524 21179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31: Bootstrap starting.
I20260619 04:38:22.492830 21179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:22.494099 21179 log.cc:824] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:22.497143 21182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:22.475682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58869 (local address 127.0.0.1:33793)
0619 04:38:22.476446 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:22.476476 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:22.476526 (+ 50us) server_negotiation.cc:407] Connection header received
0619 04:38:22.478265 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:22.478286 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:22.478681 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:22.479016 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:22.480739 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:22.482097 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:22.483935 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:22.484655 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:22.484785 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:22.489659 (+ 4874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:22.489670 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:22.489688 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:22.489730 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:22.492771 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:22.493627 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:22.493633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:22.493640 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:22.493808 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:22.494310 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:22.494315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:22.494317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:22.495033 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:22.495347 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:22.495619 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 04:38:22.496113 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":332,"threads_started":1}
I20260619 04:38:22.497187 21179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31: No bootstrap required, opened a new log
I20260619 04:38:22.500608 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:38:22.506718 21179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831b56c27efc47e1ba19d122d02b3f31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33793 } }
I20260619 04:38:22.507099 21179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:22.507186 21179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 831b56c27efc47e1ba19d122d02b3f31, State: Initialized, Role: FOLLOWER
I20260619 04:38:22.507761 21179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831b56c27efc47e1ba19d122d02b3f31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33793 } }
I20260619 04:38:22.507987 21179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:38:22.508134 21179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:38:22.508311 21179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:22.510815 21179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831b56c27efc47e1ba19d122d02b3f31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33793 } }
I20260619 04:38:22.511358 21179 leader_election.cc:302] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 831b56c27efc47e1ba19d122d02b3f31; no voters:
I20260619 04:38:22.511976 21179 leader_election.cc:288] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:38:22.512118 21189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:22.514113 21179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:22.514081 21189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31 [term 1 LEADER]: Becoming Leader. State: Replica: 831b56c27efc47e1ba19d122d02b3f31, State: Running, Role: LEADER
I20260619 04:38:22.514612 21189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831b56c27efc47e1ba19d122d02b3f31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33793 } }
I20260619 04:38:22.521294 21182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:22.505724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48055 (local address 127.0.0.1:33793)
0619 04:38:22.505924 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:22.505930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:22.506268 (+ 338us) server_negotiation.cc:407] Connection header received
0619 04:38:22.506403 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:22.506408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:22.506487 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:22.506594 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:22.507955 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:22.509067 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:22.510316 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:22.510944 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:22.511060 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:22.513881 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:22.513895 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:22.513900 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:22.513940 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:22.517969 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:22.518813 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:22.518819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:22.518823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:22.518985 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:22.519435 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:22.519441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:22.519445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:22.519848 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:22.520139 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:22.520444 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 04:38:22.520786 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:38:22.525280 21191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 831b56c27efc47e1ba19d122d02b3f31. Latest consensus state: current_term: 1 leader_uuid: "831b56c27efc47e1ba19d122d02b3f31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831b56c27efc47e1ba19d122d02b3f31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33793 } } }
I20260619 04:38:22.525704 21191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:22.526258 21190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "831b56c27efc47e1ba19d122d02b3f31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831b56c27efc47e1ba19d122d02b3f31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33793 } } }
I20260619 04:38:22.526993 21190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31 [sys.catalog]: This master's current role is: LEADER
W20260619 04:38:22.530094 21206 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 04:38:22.530192 21206 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 04:38:22.530293 21205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:22.532298 21205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:22.541347 21205 catalog_manager.cc:1347] Generated new cluster ID: 272d36bee4b24583aad38fe301954873
I20260619 04:38:22.541457 21205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:22.572252 21205 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:22.573699 21205 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:22.583981 21205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 831b56c27efc47e1ba19d122d02b3f31: Generated new TSK 0
I20260619 04:38:22.585053 21205 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:38:22.589982 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 21112
I20260619 04:38:22.591950 21174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:22.686857 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 21112
2026-06-19T04:38:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1887 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:38:22 dist-test-slave-n3tq krb5kdc[21212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:38:22 dist-test-slave-n3tq krb5kdc[21212](info): set up 2 sockets
Jun 19 04:38:22 dist-test-slave-n3tq krb5kdc[21212](info): commencing operation
krb5kdc: starting...
W20260619 04:38:23.692571 388 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:38:23 dist-test-slave-n3tq krb5kdc[21212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:38:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:38:23.924396 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34381
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:24.243854 21225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:24.244161 21225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:24.244304 21225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:24.251636 21225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:24.251778 21225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:24.251868 21225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:24.251950 21225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:24.252012 21225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:24.261919 21225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:24.263195 21225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:24.264515 21225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:24.276355 21231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:24.276384 21230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:24.276422 21233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:24.277506 21225 server_base.cc:1034] running on GCE node
I20260619 04:38:24.278348 21225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:24.279799 21225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:24.280994 21225 hybrid_clock.cc:648] HybridClock initialized: now 1781843904280948 us; error 45 us; skew 500 ppm
I20260619 04:38:24.281334 21225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:24 dist-test-slave-n3tq krb5kdc[21212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:24.284391 21225 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:38:24.285925 21225 webserver.cc:460] Webserver started at http://127.0.0.1:36555/ using document root <none> and password file <none>
I20260619 04:38:24.286590 21225 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:24.286685 21225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:24.286933 21225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:24.289954 21225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "8a9124b68ad249e0a93e954e5e17b44f"
format_stamp: "Formatted at 2026-06-19 04:38:24 on dist-test-slave-n3tq"
server_key: "939b42d8ce955294a63477676c1dd612"
server_key_iv: "29e6c8f55477f5b79cb0bb3db07d525a"
server_key_version: "encryptionkey@0"
I20260619 04:38:24.290669 21225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "8a9124b68ad249e0a93e954e5e17b44f"
format_stamp: "Formatted at 2026-06-19 04:38:24 on dist-test-slave-n3tq"
server_key: "939b42d8ce955294a63477676c1dd612"
server_key_iv: "29e6c8f55477f5b79cb0bb3db07d525a"
server_key_version: "encryptionkey@0"
I20260619 04:38:24.295801 21225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 04:38:24.299399 21240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:24.301050 21225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 04:38:24.301198 21225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "8a9124b68ad249e0a93e954e5e17b44f"
format_stamp: "Formatted at 2026-06-19 04:38:24 on dist-test-slave-n3tq"
server_key: "939b42d8ce955294a63477676c1dd612"
server_key_iv: "29e6c8f55477f5b79cb0bb3db07d525a"
server_key_version: "encryptionkey@0"
I20260619 04:38:24.301370 21225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:24.315147 21225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:24.319343 21225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:24.319581 21225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:24.336329 21225 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34381
I20260619 04:38:24.336361 21291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34381 every 8 connection(s)
I20260619 04:38:24.337772 21225 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:24.341454 21292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:24.347271 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 21225
I20260619 04:38:24.347474 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:24.347971 388 external_mini_cluster.cc:1428] Setting key b9b168f2e4bf78be8c1e5d4d4637fc38
I20260619 04:38:24.352617 21292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f: Bootstrap starting.
I20260619 04:38:24.356069 21292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:24.357347 21292 log.cc:824] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:24.360569 21292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f: No bootstrap required, opened a new log
I20260619 04:38:24.361883 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:24.349830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58679 (local address 127.0.0.1:34381)
0619 04:38:24.350705 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:24.350718 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:24.350762 (+ 44us) server_negotiation.cc:407] Connection header received
0619 04:38:24.352509 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:24.352535 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:24.352980 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:24.353312 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:24.353918 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:24.355262 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:24.357123 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:24.357952 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:24.358154 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:24.359136 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:24.359145 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:24.359164 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:24.359875 (+ 711us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:24.359971 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:24.360003 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:24.360128 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:24.360289 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 04:38:24.360597 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":429,"threads_started":1}
I20260619 04:38:24.365723 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:38:24.370639 21292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9124b68ad249e0a93e954e5e17b44f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34381 } }
I20260619 04:38:24.371135 21292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:24.371232 21292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9124b68ad249e0a93e954e5e17b44f, State: Initialized, Role: FOLLOWER
I20260619 04:38:24.371869 21292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9124b68ad249e0a93e954e5e17b44f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34381 } }
I20260619 04:38:24.372160 21292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:38:24.372287 21292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:38:24.372447 21292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:24.375130 21292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9124b68ad249e0a93e954e5e17b44f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34381 } }
I20260619 04:38:24.375771 21292 leader_election.cc:302] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a9124b68ad249e0a93e954e5e17b44f; no voters:
I20260619 04:38:24.376439 21292 leader_election.cc:288] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:38:24.376613 21302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:24.377645 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:24.370860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38749 (local address 127.0.0.1:34381)
0619 04:38:24.371076 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:24.371083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:24.371558 (+ 475us) server_negotiation.cc:407] Connection header received
0619 04:38:24.371745 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:24.371750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:24.371837 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:24.371951 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:24.372514 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:24.373553 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:24.374902 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:24.375548 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:24.375675 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:24.376799 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:24.376805 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:24.376813 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:24.376844 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:24.376909 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:24.376932 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:24.377034 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:24.377166 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 04:38:24.377273 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 04:38:24.378728 21292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:24.380731 21302 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f [term 1 LEADER]: Becoming Leader. State: Replica: 8a9124b68ad249e0a93e954e5e17b44f, State: Running, Role: LEADER
I20260619 04:38:24.381390 21302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9124b68ad249e0a93e954e5e17b44f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34381 } }
I20260619 04:38:24.383950 21303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a9124b68ad249e0a93e954e5e17b44f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9124b68ad249e0a93e954e5e17b44f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34381 } } }
I20260619 04:38:24.384346 21303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:24.384747 21304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a9124b68ad249e0a93e954e5e17b44f. Latest consensus state: current_term: 1 leader_uuid: "8a9124b68ad249e0a93e954e5e17b44f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9124b68ad249e0a93e954e5e17b44f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34381 } } }
I20260619 04:38:24.385068 21304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:24.387159 21312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:24.391371 21312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:24.400283 21312 catalog_manager.cc:1347] Generated new cluster ID: f50ba5aa6fc04056ac7194f1aef2fc83
I20260619 04:38:24.400421 21312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:24.423913 21312 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:24.425499 21312 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:24.434237 21312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a9124b68ad249e0a93e954e5e17b44f: Generated new TSK 0
I20260619 04:38:24.435144 21312 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:38:24.448720 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 21225
W20260619 04:38:24.449796 21287 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58679
W20260619 04:38:24.450519 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 21225: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:38:24.450649 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 21225
2026-06-19T04:38:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1763 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:38:24 dist-test-slave-n3tq krb5kdc[21323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:38:24 dist-test-slave-n3tq krb5kdc[21323](info): set up 2 sockets
Jun 19 04:38:24 dist-test-slave-n3tq krb5kdc[21323](info): commencing operation
krb5kdc: starting...
W20260619 04:38:25.460439 388 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:38:25 dist-test-slave-n3tq krb5kdc[21323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:38:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:38:25.699343 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39417
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:26.021756 21336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:26.022030 21336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:26.022161 21336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:26.029296 21336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:26.029402 21336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:26.029474 21336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:26.029556 21336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:26.029610 21336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:26.039230 21336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:26.040500 21336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:26.041770 21336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:26.053090 21341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:26.053148 21342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:26.053126 21344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:26.053781 21336 server_base.cc:1034] running on GCE node
I20260619 04:38:26.054543 21336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:26.056131 21336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:26.057370 21336 hybrid_clock.cc:648] HybridClock initialized: now 1781843906057319 us; error 44 us; skew 500 ppm
I20260619 04:38:26.057695 21336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:26 dist-test-slave-n3tq krb5kdc[21323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:26.069871 21336 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:38:26.071519 21336 webserver.cc:460] Webserver started at http://127.0.0.1:35903/ using document root <none> and password file <none>
I20260619 04:38:26.072202 21336 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:26.072304 21336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:26.072608 21336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:26.075570 21336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "3ea82e5b080d42d79b9e33b3f071aa67"
format_stamp: "Formatted at 2026-06-19 04:38:26 on dist-test-slave-n3tq"
server_key: "a98015ed564446e2c30e901d2c246af4"
server_key_iv: "1ac0fbb019bcfe9e460888200a0d9d3c"
server_key_version: "encryptionkey@0"
I20260619 04:38:26.076283 21336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "3ea82e5b080d42d79b9e33b3f071aa67"
format_stamp: "Formatted at 2026-06-19 04:38:26 on dist-test-slave-n3tq"
server_key: "a98015ed564446e2c30e901d2c246af4"
server_key_iv: "1ac0fbb019bcfe9e460888200a0d9d3c"
server_key_version: "encryptionkey@0"
I20260619 04:38:26.081401 21336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 04:38:26.084954 21351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:26.086405 21336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 04:38:26.086560 21336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "3ea82e5b080d42d79b9e33b3f071aa67"
format_stamp: "Formatted at 2026-06-19 04:38:26 on dist-test-slave-n3tq"
server_key: "a98015ed564446e2c30e901d2c246af4"
server_key_iv: "1ac0fbb019bcfe9e460888200a0d9d3c"
server_key_version: "encryptionkey@0"
I20260619 04:38:26.086755 21336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:26.092832 21336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:26.096935 21336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:26.097193 21336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:26.114579 21336 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39417
I20260619 04:38:26.114619 21402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39417 every 8 connection(s)
I20260619 04:38:26.115972 21336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:26.119823 21403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:26.121996 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 21336
I20260619 04:38:26.122193 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:26.122726 388 external_mini_cluster.cc:1428] Setting key 83aa3fc77c6e6cc8e924ba37060e40de
I20260619 04:38:26.130563 21406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:26.124739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35281 (local address 127.0.0.1:39417)
0619 04:38:26.125537 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:26.125553 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:26.125597 (+ 44us) server_negotiation.cc:407] Connection header received
0619 04:38:26.127197 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:26.127221 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:26.127597 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:26.127975 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:26.129020 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:26.129027 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:26.129040 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:26.129054 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:26.129117 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:26.129138 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:26.129331 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:26.129458 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 04:38:26.129642 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":341,"threads_started":1}
I20260619 04:38:26.133558 21403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67: Bootstrap starting.
I20260619 04:38:26.134042 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:38:26.136741 21403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:26.137799 21403 log.cc:824] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:26.140854 21403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67: No bootstrap required, opened a new log
I20260619 04:38:26.141149 21406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:26.138191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41895 (local address 127.0.0.1:39417)
0619 04:38:26.138373 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:26.138378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:26.139359 (+ 981us) server_negotiation.cc:407] Connection header received
0619 04:38:26.139512 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:26.139516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:26.139563 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:26.139643 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:26.140550 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:26.140556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:26.140561 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:26.140577 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:26.140615 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:26.140630 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:26.140729 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:26.140830 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 04:38:26.140896 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 04:38:26.148713 21403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea82e5b080d42d79b9e33b3f071aa67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39417 } }
I20260619 04:38:26.149160 21403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:26.149255 21403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ea82e5b080d42d79b9e33b3f071aa67, State: Initialized, Role: FOLLOWER
I20260619 04:38:26.149837 21403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea82e5b080d42d79b9e33b3f071aa67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39417 } }
I20260619 04:38:26.150075 21403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:38:26.150216 21403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:38:26.150362 21403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:26.152694 21403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea82e5b080d42d79b9e33b3f071aa67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39417 } }
I20260619 04:38:26.153213 21403 leader_election.cc:302] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ea82e5b080d42d79b9e33b3f071aa67; no voters:
I20260619 04:38:26.153769 21403 leader_election.cc:288] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:38:26.153914 21413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:26.154909 21413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67 [term 1 LEADER]: Becoming Leader. State: Replica: 3ea82e5b080d42d79b9e33b3f071aa67, State: Running, Role: LEADER
I20260619 04:38:26.155472 21413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea82e5b080d42d79b9e33b3f071aa67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39417 } }
I20260619 04:38:26.155917 21403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:26.158034 21414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ea82e5b080d42d79b9e33b3f071aa67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea82e5b080d42d79b9e33b3f071aa67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39417 } } }
I20260619 04:38:26.158370 21414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:26.159180 21420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:26.159044 21415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ea82e5b080d42d79b9e33b3f071aa67. Latest consensus state: current_term: 1 leader_uuid: "3ea82e5b080d42d79b9e33b3f071aa67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea82e5b080d42d79b9e33b3f071aa67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39417 } } }
I20260619 04:38:26.160586 21415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:26.165787 21420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:26.174867 21420 catalog_manager.cc:1347] Generated new cluster ID: a43acea24ad74afe979283517dc63267
I20260619 04:38:26.174993 21420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:26.194768 21420 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:26.195194 21420 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:26.205016 21420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ea82e5b080d42d79b9e33b3f071aa67: Generated new TSK 0
I20260619 04:38:26.205814 21420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:38:26.222604 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 21336
W20260619 04:38:26.223767 21398 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35281
W20260619 04:38:26.224447 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 21336: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:38:26.224607 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 21336
2026-06-19T04:38:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1773 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:38:26 dist-test-slave-n3tq krb5kdc[21434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:38:26 dist-test-slave-n3tq krb5kdc[21434](info): set up 2 sockets
Jun 19 04:38:26 dist-test-slave-n3tq krb5kdc[21434](info): commencing operation
krb5kdc: starting...
W20260619 04:38:27.236419 388 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:38:27 dist-test-slave-n3tq krb5kdc[21434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:38:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:38:27.465245 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33103
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:27.781904 21447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:27.782229 21447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:27.782356 21447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:27.789786 21447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:27.789919 21447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:27.790000 21447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:27.790082 21447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:27.790143 21447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:27.800154 21447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:27.801486 21447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:27.802985 21447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:27.814463 21452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:27.814426 21455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:27.814404 21453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:27.815270 21447 server_base.cc:1034] running on GCE node
I20260619 04:38:27.816020 21447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:27.817456 21447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:27.818665 21447 hybrid_clock.cc:648] HybridClock initialized: now 1781843907818616 us; error 44 us; skew 500 ppm
I20260619 04:38:27.819020 21447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:27 dist-test-slave-n3tq krb5kdc[21434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:27.821988 21447 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:38:27.823515 21447 webserver.cc:460] Webserver started at http://127.0.0.1:39601/ using document root <none> and password file <none>
I20260619 04:38:27.824191 21447 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:27.824322 21447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:27.824646 21447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:27.827598 21447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "b33326f8677a4d2c98282c7a39f3f9d5"
format_stamp: "Formatted at 2026-06-19 04:38:27 on dist-test-slave-n3tq"
server_key: "740b7625e58f06632eb749c5589d2ec5"
server_key_iv: "2b5b6be0eeaa8e5fe99682bb955c41eb"
server_key_version: "encryptionkey@0"
I20260619 04:38:27.828328 21447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "b33326f8677a4d2c98282c7a39f3f9d5"
format_stamp: "Formatted at 2026-06-19 04:38:27 on dist-test-slave-n3tq"
server_key: "740b7625e58f06632eb749c5589d2ec5"
server_key_iv: "2b5b6be0eeaa8e5fe99682bb955c41eb"
server_key_version: "encryptionkey@0"
I20260619 04:38:27.833352 21447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 04:38:27.836812 21462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:27.838061 21447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:38:27.838217 21447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "b33326f8677a4d2c98282c7a39f3f9d5"
format_stamp: "Formatted at 2026-06-19 04:38:27 on dist-test-slave-n3tq"
server_key: "740b7625e58f06632eb749c5589d2ec5"
server_key_iv: "2b5b6be0eeaa8e5fe99682bb955c41eb"
server_key_version: "encryptionkey@0"
I20260619 04:38:27.838388 21447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:27.854866 21447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:27.859012 21447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:27.859277 21447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:27.875965 21447 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33103
I20260619 04:38:27.875988 21513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33103 every 8 connection(s)
I20260619 04:38:27.877360 21447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:27.877619 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 21447
I20260619 04:38:27.877803 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:27.878338 388 external_mini_cluster.cc:1428] Setting key 5e215c0fcfa52c49049d63ef72b704ef
I20260619 04:38:27.881790 21514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 04:38:27 dist-test-slave-n3tq krb5kdc[21434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 04:38:27.892875 21514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5: Bootstrap starting.
I20260619 04:38:27.896252 21514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:27.897526 21514 log.cc:824] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:27.899916 21517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:27.880136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:44055 (local address 127.0.0.1:33103)
0619 04:38:27.880934 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:27.880946 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:27.880987 (+ 41us) server_negotiation.cc:407] Connection header received
0619 04:38:27.882654 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:27.882704 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:27.883099 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:27.883451 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:27.884879 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:27.886237 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:27.887504 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:27.888187 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:27.888360 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:27.892804 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:27.892812 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:27.892828 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:27.892868 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:27.895749 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:27.896711 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:27.896717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:27.896724 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:27.896887 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:27.897326 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:27.897329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:27.897331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:27.897880 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:27.898204 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:27.898691 (+ 487us) server_negotiation.cc:299] Negotiation successful
0619 04:38:27.898976 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":378,"threads_started":1}
I20260619 04:38:27.900420 21514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5: No bootstrap required, opened a new log
I20260619 04:38:27.903484 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:38:27.908075 21514 raft_consensus.cc:348] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33326f8677a4d2c98282c7a39f3f9d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33103 } }
I20260619 04:38:27.908571 21514 raft_consensus.cc:374] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:27.908668 21514 raft_consensus.cc:729] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b33326f8677a4d2c98282c7a39f3f9d5, State: Initialized, Role: FOLLOWER
I20260619 04:38:27.909243 21514 consensus_queue.cc:260] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33326f8677a4d2c98282c7a39f3f9d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33103 } }
I20260619 04:38:27.909451 21514 raft_consensus.cc:388] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:38:27.909574 21514 raft_consensus.cc:482] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:38:27.909718 21514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:27.911875 21514 raft_consensus.cc:504] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33326f8677a4d2c98282c7a39f3f9d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33103 } }
I20260619 04:38:27.912376 21514 leader_election.cc:302] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b33326f8677a4d2c98282c7a39f3f9d5; no voters:
I20260619 04:38:27.912978 21514 leader_election.cc:288] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:38:27.913056 21524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:27.914876 21514 sys_catalog.cc:564] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:27.915555 21524 raft_consensus.cc:686] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5 [term 1 LEADER]: Becoming Leader. State: Replica: b33326f8677a4d2c98282c7a39f3f9d5, State: Running, Role: LEADER
I20260619 04:38:27.916196 21524 consensus_queue.cc:237] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33326f8677a4d2c98282c7a39f3f9d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33103 } }
I20260619 04:38:27.920375 21526 sys_catalog.cc:455] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b33326f8677a4d2c98282c7a39f3f9d5. Latest consensus state: current_term: 1 leader_uuid: "b33326f8677a4d2c98282c7a39f3f9d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33326f8677a4d2c98282c7a39f3f9d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33103 } } }
I20260619 04:38:27.921243 21526 sys_catalog.cc:458] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:27.920950 21525 sys_catalog.cc:455] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b33326f8677a4d2c98282c7a39f3f9d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33326f8677a4d2c98282c7a39f3f9d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33103 } } }
I20260619 04:38:27.921749 21525 sys_catalog.cc:458] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:27.923512 21517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:27.907569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:43999 (local address 127.0.0.1:33103)
0619 04:38:27.907741 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:27.907747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:27.908018 (+ 271us) server_negotiation.cc:407] Connection header received
0619 04:38:27.908154 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:27.908159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:27.908235 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:27.908349 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:27.909759 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:27.910558 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:27.911776 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:27.912221 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:27.912325 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:27.915399 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:27.915405 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:27.915413 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:27.915452 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:27.919044 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:27.919986 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:27.919991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:27.919994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:27.920186 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:27.920824 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:27.920829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:27.920832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:27.921221 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:27.921574 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:27.922923 (+ 1349us) server_negotiation.cc:299] Negotiation successful
0619 04:38:27.923115 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 04:38:27.926983 21535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:27.932109 21535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:27.940658 21535 catalog_manager.cc:1347] Generated new cluster ID: 42f1d9f1e04f4eec9b50c773b131562e
I20260619 04:38:27.940794 21535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:27.949283 21535 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:27.951013 21535 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:27.967747 21535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b33326f8677a4d2c98282c7a39f3f9d5: Generated new TSK 0
I20260619 04:38:27.968782 21535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:38:27.979153 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 21447
I20260619 04:38:27.980228 21509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:28.069265 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 21447
W20260619 04:38:28.083487 21432 connection.cc:535] client connection to 127.0.0.1:33103 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33103): Connection reset by peer (error 104)
2026-06-19T04:38:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1845 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:38:28 dist-test-slave-n3tq krb5kdc[21547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:38:28 dist-test-slave-n3tq krb5kdc[21547](info): set up 2 sockets
Jun 19 04:38:28 dist-test-slave-n3tq krb5kdc[21547](info): commencing operation
krb5kdc: starting...
W20260619 04:38:29.084573 388 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:38:29 dist-test-slave-n3tq krb5kdc[21547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:38:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:38:29.323001 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42563
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:29.643079 21560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:29.643389 21560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:29.643527 21560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:29.650676 21560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:29.650804 21560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:29.650882 21560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:29.650959 21560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:29.651018 21560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:29.660792 21560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:29.662070 21560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:29.663316 21560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:29.674407 21565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:29.674376 21566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:29.674443 21568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:29.675122 21560 server_base.cc:1034] running on GCE node
I20260619 04:38:29.675813 21560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:29.677205 21560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:29.678458 21560 hybrid_clock.cc:648] HybridClock initialized: now 1781843909678411 us; error 36 us; skew 500 ppm
I20260619 04:38:29.678828 21560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:29 dist-test-slave-n3tq krb5kdc[21547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:29.682018 21560 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:38:29.683733 21560 webserver.cc:460] Webserver started at http://127.0.0.1:41049/ using document root <none> and password file <none>
I20260619 04:38:29.684495 21560 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:29.684613 21560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:29.684873 21560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:29.687769 21560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "d36fd09ca9a64e4bb6ef9f4c856f141f"
format_stamp: "Formatted at 2026-06-19 04:38:29 on dist-test-slave-n3tq"
server_key: "490cbdca054d869991e7d28559576dd9"
server_key_iv: "866414f147a2d07a523055260dc82230"
server_key_version: "encryptionkey@0"
I20260619 04:38:29.688484 21560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "d36fd09ca9a64e4bb6ef9f4c856f141f"
format_stamp: "Formatted at 2026-06-19 04:38:29 on dist-test-slave-n3tq"
server_key: "490cbdca054d869991e7d28559576dd9"
server_key_iv: "866414f147a2d07a523055260dc82230"
server_key_version: "encryptionkey@0"
I20260619 04:38:29.693403 21560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 04:38:29.696787 21575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:29.698033 21560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:38:29.698184 21560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "d36fd09ca9a64e4bb6ef9f4c856f141f"
format_stamp: "Formatted at 2026-06-19 04:38:29 on dist-test-slave-n3tq"
server_key: "490cbdca054d869991e7d28559576dd9"
server_key_iv: "866414f147a2d07a523055260dc82230"
server_key_version: "encryptionkey@0"
I20260619 04:38:29.698352 21560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:29.743971 21560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:29.748441 21560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:29.748749 21560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:29.765268 21560 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42563
I20260619 04:38:29.765287 21626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42563 every 8 connection(s)
I20260619 04:38:29.766548 21560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:29.770093 21627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:29.776821 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 21560
I20260619 04:38:29.777010 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:29.777495 388 external_mini_cluster.cc:1428] Setting key 632697e02f67acb3bbcdf8af737d47f3
I20260619 04:38:29.780340 21627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f: Bootstrap starting.
I20260619 04:38:29.783532 21627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:29.784652 21627 log.cc:824] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:29.787436 21627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f: No bootstrap required, opened a new log
I20260619 04:38:29.790799 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:29.779283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:56621 (local address 127.0.0.1:42563)
0619 04:38:29.779973 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:29.779985 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:29.780031 (+ 46us) server_negotiation.cc:407] Connection header received
0619 04:38:29.781784 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:29.781806 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:29.782201 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:29.782493 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:29.783069 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:29.784364 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:29.785978 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:29.786651 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:29.786791 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:29.788078 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:29.788090 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:29.788107 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:29.788819 (+ 712us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:29.788909 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:29.788942 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:29.789106 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:29.789346 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 04:38:29.789517 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":300,"threads_started":1}
I20260619 04:38:29.794369 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:38:29.796383 21627 raft_consensus.cc:348] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36fd09ca9a64e4bb6ef9f4c856f141f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42563 } }
I20260619 04:38:29.796849 21627 raft_consensus.cc:374] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:29.796952 21627 raft_consensus.cc:729] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36fd09ca9a64e4bb6ef9f4c856f141f, State: Initialized, Role: FOLLOWER
I20260619 04:38:29.797610 21627 consensus_queue.cc:260] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36fd09ca9a64e4bb6ef9f4c856f141f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42563 } }
I20260619 04:38:29.797916 21627 raft_consensus.cc:388] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:38:29.798040 21627 raft_consensus.cc:482] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:38:29.798236 21627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:29.801144 21627 raft_consensus.cc:504] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36fd09ca9a64e4bb6ef9f4c856f141f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42563 } }
I20260619 04:38:29.801715 21627 leader_election.cc:302] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d36fd09ca9a64e4bb6ef9f4c856f141f; no voters:
I20260619 04:38:29.802306 21627 leader_election.cc:288] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:38:29.802428 21637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:29.804575 21627 sys_catalog.cc:564] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:29.806401 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:29.799752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:45609 (local address 127.0.0.1:42563)
0619 04:38:29.799925 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:29.799931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:29.800423 (+ 492us) server_negotiation.cc:407] Connection header received
0619 04:38:29.800595 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:29.800600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:29.800662 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:29.800736 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:29.801349 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:29.802264 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:29.803981 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:29.804529 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:29.804635 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:29.805678 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:29.805684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:29.805688 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:29.805704 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:29.805762 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:29.805781 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:29.805875 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:29.806046 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 04:38:29.806108 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 04:38:29.809007 21637 raft_consensus.cc:686] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f [term 1 LEADER]: Becoming Leader. State: Replica: d36fd09ca9a64e4bb6ef9f4c856f141f, State: Running, Role: LEADER
I20260619 04:38:29.809650 21637 consensus_queue.cc:237] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36fd09ca9a64e4bb6ef9f4c856f141f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42563 } }
I20260619 04:38:29.812642 21638 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d36fd09ca9a64e4bb6ef9f4c856f141f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36fd09ca9a64e4bb6ef9f4c856f141f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42563 } } }
I20260619 04:38:29.812731 21639 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d36fd09ca9a64e4bb6ef9f4c856f141f. Latest consensus state: current_term: 1 leader_uuid: "d36fd09ca9a64e4bb6ef9f4c856f141f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36fd09ca9a64e4bb6ef9f4c856f141f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42563 } } }
I20260619 04:38:29.813104 21638 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:29.813180 21639 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:29.815083 21644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:29.820816 21644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:29.829813 21644 catalog_manager.cc:1347] Generated new cluster ID: 45c0c92d254e49f7bee92455f1d718e5
I20260619 04:38:29.829936 21644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:29.843329 21644 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:29.845146 21644 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:29.859462 21644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d36fd09ca9a64e4bb6ef9f4c856f141f: Generated new TSK 0
I20260619 04:38:29.860544 21644 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:38:29.868876 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 21560
W20260619 04:38:29.869963 21622 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.21:56621
W20260619 04:38:29.870671 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 21560: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:38:29.870831 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 21560
W20260619 04:38:29.884117 21545 connection.cc:535] client connection to 127.0.0.1:42563 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42563): Connection reset by peer (error 104)
2026-06-19T04:38:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1800 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:38:29 dist-test-slave-n3tq krb5kdc[21658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:38:29 dist-test-slave-n3tq krb5kdc[21658](info): set up 2 sockets
Jun 19 04:38:29 dist-test-slave-n3tq krb5kdc[21658](info): commencing operation
krb5kdc: starting...
W20260619 04:38:30.884562 388 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:38:31 dist-test-slave-n3tq krb5kdc[21658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:38:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:38:31.117368 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:31.436748 21671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:31.437028 21671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:31.437162 21671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:31.444274 21671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:31.444384 21671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:31.444438 21671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:31.444536 21671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:31.444597 21671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:31.454243 21671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:31.455475 21671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:31.456770 21671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:31.467465 21676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:31.467500 21679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:31.467554 21677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:31.468324 21671 server_base.cc:1034] running on GCE node
I20260619 04:38:31.469038 21671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:31.470312 21671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:31.471500 21671 hybrid_clock.cc:648] HybridClock initialized: now 1781843911471452 us; error 38 us; skew 500 ppm
I20260619 04:38:31.471797 21671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:31 dist-test-slave-n3tq krb5kdc[21658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:31.482906 21671 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:38:31.484427 21671 webserver.cc:460] Webserver started at http://127.0.0.1:36529/ using document root <none> and password file <none>
I20260619 04:38:31.485131 21671 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:31.485245 21671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:31.485525 21671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:31.488490 21671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "4c399fafb2ab420987ec349ce8cb67f2"
format_stamp: "Formatted at 2026-06-19 04:38:31 on dist-test-slave-n3tq"
server_key: "89f65581ffdcb9bcbca96a3ad46f7838"
server_key_iv: "da6ddfd27171b4178891fb9422c77a91"
server_key_version: "encryptionkey@0"
I20260619 04:38:31.489231 21671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "4c399fafb2ab420987ec349ce8cb67f2"
format_stamp: "Formatted at 2026-06-19 04:38:31 on dist-test-slave-n3tq"
server_key: "89f65581ffdcb9bcbca96a3ad46f7838"
server_key_iv: "da6ddfd27171b4178891fb9422c77a91"
server_key_version: "encryptionkey@0"
I20260619 04:38:31.494242 21671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:38:31.497736 21686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:31.499106 21671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:38:31.499289 21671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "4c399fafb2ab420987ec349ce8cb67f2"
format_stamp: "Formatted at 2026-06-19 04:38:31 on dist-test-slave-n3tq"
server_key: "89f65581ffdcb9bcbca96a3ad46f7838"
server_key_iv: "da6ddfd27171b4178891fb9422c77a91"
server_key_version: "encryptionkey@0"
I20260619 04:38:31.499481 21671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:31.505084 21671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:31.509194 21671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:31.509457 21671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:31.526379 21671 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44705
I20260619 04:38:31.526432 21737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44705 every 8 connection(s)
I20260619 04:38:31.527719 21671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:31.530102 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 21671
I20260619 04:38:31.530261 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:31.530694 388 external_mini_cluster.cc:1428] Setting key a3dc7fabd5f6939696834010fe455212
I20260619 04:38:31.531869 21738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:31.538903 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:31.532401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:39327 (local address 127.0.0.1:44705)
0619 04:38:31.533208 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:31.533223 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:31.533272 (+ 49us) server_negotiation.cc:407] Connection header received
0619 04:38:31.535102 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:31.535129 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:31.535577 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:31.536026 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:31.537032 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:31.537041 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:31.537063 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:31.537078 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:31.537159 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:31.537188 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:31.537402 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:31.537525 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 04:38:31.537724 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":310,"threads_started":1}
I20260619 04:38:31.542326 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:38:31.545846 21738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2: Bootstrap starting.
I20260619 04:38:31.549602 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:31.547491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:37333 (local address 127.0.0.1:44705)
0619 04:38:31.547689 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:31.547695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:31.547899 (+ 204us) server_negotiation.cc:407] Connection header received
0619 04:38:31.548034 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:31.548040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:31.548102 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:31.548207 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:31.549013 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:31.549019 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:31.549023 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:31.549041 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:31.549085 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:31.549106 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:31.549219 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:31.549262 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 04:38:31.549319 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 04:38:31.549607 21738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:31.550750 21738 log.cc:824] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:31.553450 21738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2: No bootstrap required, opened a new log
I20260619 04:38:31.560580 21738 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c399fafb2ab420987ec349ce8cb67f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44705 } }
I20260619 04:38:31.560974 21738 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:31.561085 21738 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c399fafb2ab420987ec349ce8cb67f2, State: Initialized, Role: FOLLOWER
I20260619 04:38:31.561614 21738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c399fafb2ab420987ec349ce8cb67f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44705 } }
I20260619 04:38:31.561825 21738 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:38:31.561950 21738 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:38:31.562100 21738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:31.564172 21738 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c399fafb2ab420987ec349ce8cb67f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44705 } }
I20260619 04:38:31.564658 21738 leader_election.cc:302] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c399fafb2ab420987ec349ce8cb67f2; no voters:
I20260619 04:38:31.565227 21738 leader_election.cc:288] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:38:31.565342 21748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:31.565943 21748 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2 [term 1 LEADER]: Becoming Leader. State: Replica: 4c399fafb2ab420987ec349ce8cb67f2, State: Running, Role: LEADER
I20260619 04:38:31.566433 21748 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c399fafb2ab420987ec349ce8cb67f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44705 } }
I20260619 04:38:31.567190 21738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:31.569751 21750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c399fafb2ab420987ec349ce8cb67f2. Latest consensus state: current_term: 1 leader_uuid: "4c399fafb2ab420987ec349ce8cb67f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c399fafb2ab420987ec349ce8cb67f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44705 } } }
I20260619 04:38:31.570169 21750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:31.570945 21749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c399fafb2ab420987ec349ce8cb67f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c399fafb2ab420987ec349ce8cb67f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44705 } } }
I20260619 04:38:31.571293 21749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:31.571803 21757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:31.576808 21757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:31.584638 21757 catalog_manager.cc:1347] Generated new cluster ID: 425a19268fb247938b8c44bcf3ed6b70
I20260619 04:38:31.584745 21757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:31.608317 21757 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:31.608850 21757 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:31.616173 21757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c399fafb2ab420987ec349ce8cb67f2: Generated new TSK 0
I20260619 04:38:31.617007 21757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:38:31.629402 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 21671
W20260619 04:38:31.630614 21733 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.21:39327
W20260619 04:38:31.631289 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 21671: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:38:31.631435 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 21671
2026-06-19T04:38:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1760 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:38:31 dist-test-slave-n3tq krb5kdc[21769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:38:31 dist-test-slave-n3tq krb5kdc[21769](info): set up 2 sockets
Jun 19 04:38:31 dist-test-slave-n3tq krb5kdc[21769](info): commencing operation
krb5kdc: starting...
W20260619 04:38:32.648667 388 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:38:32 dist-test-slave-n3tq krb5kdc[21769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:38:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:38:32.875015 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43771
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:33.185549 21782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:33.185818 21782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:33.185937 21782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:33.192956 21782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:33.193058 21782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:33.193126 21782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:33.193208 21782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:33.193264 21782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:33.202836 21782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43771
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:33.204102 21782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:33.205390 21782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:33.216612 21787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:33.216645 21788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:33.216529 21790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:33.217662 21782 server_base.cc:1034] running on GCE node
I20260619 04:38:33.218377 21782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:33.219709 21782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:33.220892 21782 hybrid_clock.cc:648] HybridClock initialized: now 1781843913220853 us; error 33 us; skew 500 ppm
I20260619 04:38:33.221213 21782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:33 dist-test-slave-n3tq krb5kdc[21769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:33.224136 21782 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:38:33.225535 21782 webserver.cc:460] Webserver started at http://127.0.0.1:43559/ using document root <none> and password file <none>
I20260619 04:38:33.226207 21782 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:33.226318 21782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:33.226610 21782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:33.229521 21782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "3c478f2419f741bda1dced10fdfd64e7"
format_stamp: "Formatted at 2026-06-19 04:38:33 on dist-test-slave-n3tq"
server_key: "923939093060753abaf1e2175360d520"
server_key_iv: "0cfaa36c48ad86111be0a815803c4de9"
server_key_version: "encryptionkey@0"
I20260619 04:38:33.230257 21782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "3c478f2419f741bda1dced10fdfd64e7"
format_stamp: "Formatted at 2026-06-19 04:38:33 on dist-test-slave-n3tq"
server_key: "923939093060753abaf1e2175360d520"
server_key_iv: "0cfaa36c48ad86111be0a815803c4de9"
server_key_version: "encryptionkey@0"
I20260619 04:38:33.235535 21782 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:38:33.239117 21797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:33.240654 21782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:38:33.240819 21782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "3c478f2419f741bda1dced10fdfd64e7"
format_stamp: "Formatted at 2026-06-19 04:38:33 on dist-test-slave-n3tq"
server_key: "923939093060753abaf1e2175360d520"
server_key_iv: "0cfaa36c48ad86111be0a815803c4de9"
server_key_version: "encryptionkey@0"
I20260619 04:38:33.240999 21782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:33.256592 21782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:33.260895 21782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:33.261165 21782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:33.277974 21782 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43771
I20260619 04:38:33.277992 21848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43771 every 8 connection(s)
I20260619 04:38:33.279323 21782 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:33.283051 21849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:33.286387 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 21782
I20260619 04:38:33.286580 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:33.286983 388 external_mini_cluster.cc:1428] Setting key b81313231a4a5f1090dbc83d794aff0a
I20260619 04:38:33.295995 21849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7: Bootstrap starting.
I20260619 04:38:33.299566 21849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:33.300566 21849 log.cc:824] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7: Log is configured to *not* fsync() on all Append() calls
Jun 19 04:38:33 dist-test-slave-n3tq krb5kdc[21769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 04:38:33.303563 21849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7: No bootstrap required, opened a new log
I20260619 04:38:33.311395 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:33.288837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:57035 (local address 127.0.0.1:43771)
0619 04:38:33.289594 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:33.289611 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:33.289672 (+ 61us) server_negotiation.cc:407] Connection header received
0619 04:38:33.291753 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:33.291785 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:33.292377 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:33.292868 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:33.294147 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:33.296085 (+ 1938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:33.297475 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:33.298497 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:33.298688 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:33.302537 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:33.302548 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:33.302567 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:33.302621 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:33.306486 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:33.307342 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:33.307351 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:33.307361 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:33.307575 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:33.308051 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:33.308056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:33.308060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:33.308795 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:33.309132 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:33.309570 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 04:38:33.309975 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":304,"threads_started":1}
I20260619 04:38:33.313795 21849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c478f2419f741bda1dced10fdfd64e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43771 } }
I20260619 04:38:33.314177 21849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:33.314270 21849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c478f2419f741bda1dced10fdfd64e7, State: Initialized, Role: FOLLOWER
I20260619 04:38:33.314862 21849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c478f2419f741bda1dced10fdfd64e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43771 } }
I20260619 04:38:33.315164 21849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:38:33.315308 21849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:38:33.315474 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:38:33.315474 21849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:33.317665 21849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c478f2419f741bda1dced10fdfd64e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43771 } }
I20260619 04:38:33.318149 21849 leader_election.cc:302] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c478f2419f741bda1dced10fdfd64e7; no voters:
I20260619 04:38:33.318744 21849 leader_election.cc:288] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:38:33.318856 21858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:33.321024 21858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7 [term 1 LEADER]: Becoming Leader. State: Replica: 3c478f2419f741bda1dced10fdfd64e7, State: Running, Role: LEADER
I20260619 04:38:33.321591 21858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c478f2419f741bda1dced10fdfd64e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43771 } }
I20260619 04:38:33.322270 21849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:33.324301 21860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c478f2419f741bda1dced10fdfd64e7. Latest consensus state: current_term: 1 leader_uuid: "3c478f2419f741bda1dced10fdfd64e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c478f2419f741bda1dced10fdfd64e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43771 } } }
I20260619 04:38:33.324720 21860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:33.333225 21859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c478f2419f741bda1dced10fdfd64e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c478f2419f741bda1dced10fdfd64e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43771 } } }
I20260619 04:38:33.333807 21859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:33.334558 21864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:33.336613 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:33.320343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:34383 (local address 127.0.0.1:43771)
0619 04:38:33.320555 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:33.320575 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:33.320593 (+ 18us) server_negotiation.cc:407] Connection header received
0619 04:38:33.320651 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:33.320656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:33.320729 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:33.320812 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:33.322591 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:33.323572 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:33.325395 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:33.326120 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:33.326243 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:33.329757 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:33.329765 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:33.329770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:33.329814 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:33.333373 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:33.334267 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:33.334272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:33.334275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:33.334411 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:33.334971 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:33.334976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:33.334979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:33.335422 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:33.335695 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:33.336052 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 04:38:33.336221 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 04:38:33.340379 21864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:33.349401 21864 catalog_manager.cc:1347] Generated new cluster ID: 9ab1868a8f0b48f99641706010a2b616
I20260619 04:38:33.349514 21864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:33.371142 21864 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:33.372368 21864 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:33.398020 21864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c478f2419f741bda1dced10fdfd64e7: Generated new TSK 0
I20260619 04:38:33.398891 21864 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:38:33.409847 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 21782
I20260619 04:38:33.411074 21844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:33.504385 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 21782
W20260619 04:38:33.518272 21767 connection.cc:535] client connection to 127.0.0.1:43771 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43771): Connection reset by peer (error 104)
2026-06-19T04:38:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1873 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:38:33 dist-test-slave-n3tq krb5kdc[21882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:38:33 dist-test-slave-n3tq krb5kdc[21882](info): set up 2 sockets
Jun 19 04:38:33 dist-test-slave-n3tq krb5kdc[21882](info): commencing operation
krb5kdc: starting...
W20260619 04:38:34.512498 388 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:38:34 dist-test-slave-n3tq krb5kdc[21882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:38:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:38:34.744335 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35717
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:35.063295 21895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:35.063593 21895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:35.063694 21895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:35.070812 21895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:35.070924 21895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:35.070976 21895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:35.071033 21895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:35.071076 21895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:35.080717 21895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35717
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:35.081923 21895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:35.083180 21895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:35.094578 21903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:35.094646 21900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:35.094671 21901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:35.095284 21895 server_base.cc:1034] running on GCE node
I20260619 04:38:35.096069 21895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:35.097522 21895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:35.098734 21895 hybrid_clock.cc:648] HybridClock initialized: now 1781843915098686 us; error 42 us; skew 500 ppm
I20260619 04:38:35.099069 21895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:35 dist-test-slave-n3tq krb5kdc[21882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:35.102178 21895 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:38:35.103632 21895 webserver.cc:460] Webserver started at http://127.0.0.1:41395/ using document root <none> and password file <none>
I20260619 04:38:35.104302 21895 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:35.104418 21895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:35.104727 21895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:35.107645 21895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "b6dec98c6a7a4628baa4d6d672811f9f"
format_stamp: "Formatted at 2026-06-19 04:38:35 on dist-test-slave-n3tq"
server_key: "9d0c5a0a0c7d1cf2ca850d361ffa8e82"
server_key_iv: "b05261bdab0e3f765b62e9619a276982"
server_key_version: "encryptionkey@0"
I20260619 04:38:35.108352 21895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "b6dec98c6a7a4628baa4d6d672811f9f"
format_stamp: "Formatted at 2026-06-19 04:38:35 on dist-test-slave-n3tq"
server_key: "9d0c5a0a0c7d1cf2ca850d361ffa8e82"
server_key_iv: "b05261bdab0e3f765b62e9619a276982"
server_key_version: "encryptionkey@0"
I20260619 04:38:35.113497 21895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 04:38:35.117022 21910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:35.118371 21895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:38:35.118559 21895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "b6dec98c6a7a4628baa4d6d672811f9f"
format_stamp: "Formatted at 2026-06-19 04:38:35 on dist-test-slave-n3tq"
server_key: "9d0c5a0a0c7d1cf2ca850d361ffa8e82"
server_key_iv: "b05261bdab0e3f765b62e9619a276982"
server_key_version: "encryptionkey@0"
I20260619 04:38:35.118743 21895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:35.153610 21895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:35.157791 21895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:35.158067 21895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:35.174289 21895 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35717
I20260619 04:38:35.174314 21961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35717 every 8 connection(s)
I20260619 04:38:35.175668 21895 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:35.176133 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 21895
I20260619 04:38:35.176301 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:35.176869 388 external_mini_cluster.cc:1428] Setting key b7267020265736d8e0af271c35d0a4a8
I20260619 04:38:35.179822 21962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:35.189366 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:35.178656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:33741 (local address 127.0.0.1:35717)
0619 04:38:35.179389 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:35.179402 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:35.179473 (+ 71us) server_negotiation.cc:407] Connection header received
0619 04:38:35.181181 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:35.181204 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:35.181599 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:35.181950 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:35.182481 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:35.183766 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:35.185046 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:35.185708 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:35.185865 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:35.187218 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:35.187225 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:35.187239 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:35.187780 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:35.187854 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:35.187877 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:35.188034 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:35.188274 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 04:38:35.188418 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":313,"threads_started":1}
I20260619 04:38:35.190006 21962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f: Bootstrap starting.
I20260619 04:38:35.193109 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:38:35.193405 21962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:35.194324 21962 log.cc:824] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:35.197032 21962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f: No bootstrap required, opened a new log
I20260619 04:38:35.203075 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:35.197241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:52765 (local address 127.0.0.1:35717)
0619 04:38:35.197371 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:35.197375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:35.197822 (+ 447us) server_negotiation.cc:407] Connection header received
0619 04:38:35.198004 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:35.198007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:35.198064 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:35.198137 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:35.198638 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:35.199422 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:35.200747 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:35.201227 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:35.201348 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:35.202405 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:35.202409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:35.202413 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:35.202430 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:35.202465 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:35.202480 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:35.202576 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:35.202762 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 04:38:35.202816 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 04:38:35.204773 21962 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dec98c6a7a4628baa4d6d672811f9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } }
I20260619 04:38:35.205108 21962 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:35.205199 21962 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6dec98c6a7a4628baa4d6d672811f9f, State: Initialized, Role: FOLLOWER
I20260619 04:38:35.205739 21962 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dec98c6a7a4628baa4d6d672811f9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } }
I20260619 04:38:35.205945 21962 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:38:35.206069 21962 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:38:35.206213 21962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:35.208220 21962 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dec98c6a7a4628baa4d6d672811f9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } }
I20260619 04:38:35.208715 21962 leader_election.cc:302] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6dec98c6a7a4628baa4d6d672811f9f; no voters:
I20260619 04:38:35.209213 21962 leader_election.cc:288] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:38:35.209329 21972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:35.209931 21972 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f [term 1 LEADER]: Becoming Leader. State: Replica: b6dec98c6a7a4628baa4d6d672811f9f, State: Running, Role: LEADER
I20260619 04:38:35.210477 21972 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dec98c6a7a4628baa4d6d672811f9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } }
I20260619 04:38:35.211206 21962 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:35.213114 21974 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6dec98c6a7a4628baa4d6d672811f9f. Latest consensus state: current_term: 1 leader_uuid: "b6dec98c6a7a4628baa4d6d672811f9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dec98c6a7a4628baa4d6d672811f9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } } }
I20260619 04:38:35.213316 21973 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6dec98c6a7a4628baa4d6d672811f9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dec98c6a7a4628baa4d6d672811f9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } } }
I20260619 04:38:35.213544 21974 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:35.213609 21973 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:35.215504 21980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:35.219964 21980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:35.228279 21980 catalog_manager.cc:1347] Generated new cluster ID: 5f641068285e496cae056c2bc267985c
I20260619 04:38:35.228391 21980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:35.249737 21980 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:35.251595 21980 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:35.261729 21980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6dec98c6a7a4628baa4d6d672811f9f: Generated new TSK 0
I20260619 04:38:35.262534 21980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:38:35.272395 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 21895
W20260619 04:38:35.273563 21957 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.21:33741
W20260619 04:38:35.274245 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 21895: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:38:35.274391 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 21895
W20260619 04:38:35.288272 21880 connection.cc:535] client connection to 127.0.0.1:35717 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35717): Connection reset by peer (error 104)
2026-06-19T04:38:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1770 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:38:35 dist-test-slave-n3tq krb5kdc[21993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:38:35 dist-test-slave-n3tq krb5kdc[21993](info): set up 2 sockets
Jun 19 04:38:35 dist-test-slave-n3tq krb5kdc[21993](info): commencing operation
krb5kdc: starting...
W20260619 04:38:36.292685 388 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:38:36 dist-test-slave-n3tq krb5kdc[21993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:38:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:38:36.519214 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39961
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:36.849681 22006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:36.850001 22006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:36.850147 22006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:36.857402 22006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:36.857535 22006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:36.857615 22006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:36.857693 22006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:36.857753 22006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:36.867592 22006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39961
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:36.868949 22006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:36.870262 22006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:36.882081 22011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:36.882121 22014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:36.882143 22012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:36.882899 22006 server_base.cc:1034] running on GCE node
I20260619 04:38:36.883638 22006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:36.885115 22006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:36.886313 22006 hybrid_clock.cc:648] HybridClock initialized: now 1781843916886270 us; error 37 us; skew 500 ppm
I20260619 04:38:36.886653 22006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:36 dist-test-slave-n3tq krb5kdc[21993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:36.889554 22006 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:38:36.890962 22006 webserver.cc:460] Webserver started at http://127.0.0.1:38455/ using document root <none> and password file <none>
I20260619 04:38:36.891620 22006 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:36.891733 22006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:36.892006 22006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:36.894955 22006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "b2faf050ba164e0e8b683d2bdafc37bf"
format_stamp: "Formatted at 2026-06-19 04:38:36 on dist-test-slave-n3tq"
server_key: "0e8b97e932ccdc9812af1de8faa6e6db"
server_key_iv: "53aaf8296ad0a3d9225d5ce18e218142"
server_key_version: "encryptionkey@0"
I20260619 04:38:36.895675 22006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "b2faf050ba164e0e8b683d2bdafc37bf"
format_stamp: "Formatted at 2026-06-19 04:38:36 on dist-test-slave-n3tq"
server_key: "0e8b97e932ccdc9812af1de8faa6e6db"
server_key_iv: "53aaf8296ad0a3d9225d5ce18e218142"
server_key_version: "encryptionkey@0"
I20260619 04:38:36.900720 22006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 04:38:36.903954 22021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:36.905277 22006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:38:36.905434 22006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "b2faf050ba164e0e8b683d2bdafc37bf"
format_stamp: "Formatted at 2026-06-19 04:38:36 on dist-test-slave-n3tq"
server_key: "0e8b97e932ccdc9812af1de8faa6e6db"
server_key_iv: "53aaf8296ad0a3d9225d5ce18e218142"
server_key_version: "encryptionkey@0"
I20260619 04:38:36.905635 22006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:36.911095 22006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:36.914759 22006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:36.915021 22006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:36.931679 22006 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39961
I20260619 04:38:36.931715 22072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39961 every 8 connection(s)
I20260619 04:38:36.933109 22006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:36.936795 22073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:36.942102 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 22006
I20260619 04:38:36.942250 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:36.942672 388 external_mini_cluster.cc:1428] Setting key 24a1bdc318e6f6b2388537c2d08cccf1
I20260619 04:38:36.947324 22073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf: Bootstrap starting.
I20260619 04:38:36.950294 22076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:36.944492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:48829 (local address 127.0.0.1:39961)
0619 04:38:36.945305 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:36.945316 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:36.945360 (+ 44us) server_negotiation.cc:407] Connection header received
0619 04:38:36.946993 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:36.947016 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:36.947374 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:36.947737 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:36.948793 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:36.948800 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:36.948815 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:36.948831 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:36.948897 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:36.948918 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:36.949116 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:36.949219 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 04:38:36.949398 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":364,"threads_started":1}
I20260619 04:38:36.950809 22073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:36.951994 22073 log.cc:824] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:36.953893 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:38:36.954648 22073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf: No bootstrap required, opened a new log
I20260619 04:38:36.959997 22076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:36.957952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:52819 (local address 127.0.0.1:39961)
0619 04:38:36.958108 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:36.958113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:36.958383 (+ 270us) server_negotiation.cc:407] Connection header received
0619 04:38:36.958494 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:36.958499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:36.958556 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:36.958657 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:36.959426 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:36.959431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:36.959435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:36.959452 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:36.959500 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:36.959519 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:36.959637 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:36.959681 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 04:38:36.959737 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 04:38:36.962459 22073 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2faf050ba164e0e8b683d2bdafc37bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39961 } }
I20260619 04:38:36.962772 22073 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:36.962846 22073 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2faf050ba164e0e8b683d2bdafc37bf, State: Initialized, Role: FOLLOWER
I20260619 04:38:36.963325 22073 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2faf050ba164e0e8b683d2bdafc37bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39961 } }
I20260619 04:38:36.963523 22073 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:38:36.963615 22073 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:38:36.963733 22073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:36.965811 22073 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2faf050ba164e0e8b683d2bdafc37bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39961 } }
I20260619 04:38:36.966324 22073 leader_election.cc:302] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2faf050ba164e0e8b683d2bdafc37bf; no voters:
I20260619 04:38:36.966815 22073 leader_election.cc:288] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:38:36.966935 22083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:36.967527 22083 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf [term 1 LEADER]: Becoming Leader. State: Replica: b2faf050ba164e0e8b683d2bdafc37bf, State: Running, Role: LEADER
I20260619 04:38:36.967973 22083 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2faf050ba164e0e8b683d2bdafc37bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39961 } }
I20260619 04:38:36.968713 22073 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:36.970516 22085 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2faf050ba164e0e8b683d2bdafc37bf. Latest consensus state: current_term: 1 leader_uuid: "b2faf050ba164e0e8b683d2bdafc37bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2faf050ba164e0e8b683d2bdafc37bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39961 } } }
I20260619 04:38:36.970933 22085 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:36.971624 22084 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2faf050ba164e0e8b683d2bdafc37bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2faf050ba164e0e8b683d2bdafc37bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39961 } } }
I20260619 04:38:36.971982 22084 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:36.972239 22090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:36.977131 22090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:36.985787 22090 catalog_manager.cc:1347] Generated new cluster ID: b654bc6fd2824c74a851c0a7baf705b9
I20260619 04:38:36.985908 22090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:37.014504 22090 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:37.014967 22090 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:37.021898 22090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2faf050ba164e0e8b683d2bdafc37bf: Generated new TSK 0
I20260619 04:38:37.022665 22090 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:38:37.032804 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 22006
W20260619 04:38:37.033869 22068 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.21:48829
W20260619 04:38:37.034632 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 22006: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:38:37.034762 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 22006
2026-06-19T04:38:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1760 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:38:37 dist-test-slave-n3tq krb5kdc[22104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:38:37 dist-test-slave-n3tq krb5kdc[22104](info): set up 2 sockets
Jun 19 04:38:37 dist-test-slave-n3tq krb5kdc[22104](info): commencing operation
krb5kdc: starting...
W20260619 04:38:38.053007 388 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:38:38 dist-test-slave-n3tq krb5kdc[22104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:38:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:38:38.287163 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40013
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:38.598734 22117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:38.599005 22117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:38.599130 22117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:38.607388 22117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:38.607529 22117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:38.607607 22117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:38.607692 22117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:38.607750 22117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:38.618229 22117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40013
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:38.619378 22117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:38.620599 22117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:38.631454 22122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:38.631513 22123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:38.631536 22125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:38.633771 22117 server_base.cc:1034] running on GCE node
I20260619 04:38:38.634563 22117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:38.635957 22117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:38.637161 22117 hybrid_clock.cc:648] HybridClock initialized: now 1781843918637119 us; error 37 us; skew 500 ppm
I20260619 04:38:38.637498 22117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:38 dist-test-slave-n3tq krb5kdc[22104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:38.640400 22117 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:38:38.641841 22117 webserver.cc:460] Webserver started at http://127.0.97.62:45225/ using document root <none> and password file <none>
I20260619 04:38:38.642498 22117 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:38.642591 22117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:38.642823 22117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:38.645769 22117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "f79eb86ba2fc46258d7ce18aea8ce88c"
format_stamp: "Formatted at 2026-06-19 04:38:38 on dist-test-slave-n3tq"
server_key: "76e5360c4ee41ef5eabd41bccdd4b301"
server_key_iv: "854b7e39e176a7b763f58343f6018618"
server_key_version: "encryptionkey@0"
I20260619 04:38:38.646471 22117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "f79eb86ba2fc46258d7ce18aea8ce88c"
format_stamp: "Formatted at 2026-06-19 04:38:38 on dist-test-slave-n3tq"
server_key: "76e5360c4ee41ef5eabd41bccdd4b301"
server_key_iv: "854b7e39e176a7b763f58343f6018618"
server_key_version: "encryptionkey@0"
I20260619 04:38:38.651376 22117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 04:38:38.654789 22132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:38.656076 22117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 04:38:38.656212 22117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "f79eb86ba2fc46258d7ce18aea8ce88c"
format_stamp: "Formatted at 2026-06-19 04:38:38 on dist-test-slave-n3tq"
server_key: "76e5360c4ee41ef5eabd41bccdd4b301"
server_key_iv: "854b7e39e176a7b763f58343f6018618"
server_key_version: "encryptionkey@0"
I20260619 04:38:38.656385 22117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:38.674846 22117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:38.678951 22117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:38.679184 22117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:38.696108 22117 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40013
I20260619 04:38:38.696161 22183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40013 every 8 connection(s)
I20260619 04:38:38.697468 22117 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:38.698350 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 22117
I20260619 04:38:38.698555 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:38.699054 388 external_mini_cluster.cc:1428] Setting key 5ccf1c2664ce34dfc0976b96e7fe992b
I20260619 04:38:38.701846 22184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 04:38:38 dist-test-slave-n3tq krb5kdc[22104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:38:38.714500 22184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c: Bootstrap starting.
I20260619 04:38:38.717626 22184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:38.718837 22184 log.cc:824] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:38.721441 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:38.700815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39337 (local address 127.0.97.62:40013)
0619 04:38:38.701588 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:38.701599 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:38.701638 (+ 39us) server_negotiation.cc:407] Connection header received
0619 04:38:38.703216 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:38.703237 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:38.703617 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:38.703919 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:38.705647 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:38.707133 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:38.708909 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:38.709585 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:38.709707 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:38.714448 (+ 4741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:38.714456 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:38.714470 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:38.714513 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:38.717389 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:38.718259 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:38.718265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:38.718271 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:38.718426 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:38.718866 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:38.718869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:38.718871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:38.719450 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:38.719766 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:38.720175 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 04:38:38.720500 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":341,"threads_started":1}
I20260619 04:38:38.721742 22184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c: No bootstrap required, opened a new log
I20260619 04:38:38.725010 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:38:38.730964 22184 raft_consensus.cc:348] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79eb86ba2fc46258d7ce18aea8ce88c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40013 } }
I20260619 04:38:38.731459 22184 raft_consensus.cc:374] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:38.731563 22184 raft_consensus.cc:729] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f79eb86ba2fc46258d7ce18aea8ce88c, State: Initialized, Role: FOLLOWER
I20260619 04:38:38.732244 22184 consensus_queue.cc:260] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79eb86ba2fc46258d7ce18aea8ce88c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40013 } }
I20260619 04:38:38.732542 22184 raft_consensus.cc:388] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:38:38.732690 22184 raft_consensus.cc:482] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:38:38.732890 22184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:38.735674 22184 raft_consensus.cc:504] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79eb86ba2fc46258d7ce18aea8ce88c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40013 } }
I20260619 04:38:38.736258 22184 leader_election.cc:302] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f79eb86ba2fc46258d7ce18aea8ce88c; no voters:
I20260619 04:38:38.736892 22184 leader_election.cc:288] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:38:38.736970 22194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:38.738155 22194 raft_consensus.cc:686] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c [term 1 LEADER]: Becoming Leader. State: Replica: f79eb86ba2fc46258d7ce18aea8ce88c, State: Running, Role: LEADER
I20260619 04:38:38.738783 22194 consensus_queue.cc:237] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79eb86ba2fc46258d7ce18aea8ce88c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40013 } }
I20260619 04:38:38.739181 22184 sys_catalog.cc:564] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:38.741902 22196 sys_catalog.cc:455] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f79eb86ba2fc46258d7ce18aea8ce88c. Latest consensus state: current_term: 1 leader_uuid: "f79eb86ba2fc46258d7ce18aea8ce88c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79eb86ba2fc46258d7ce18aea8ce88c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40013 } } }
I20260619 04:38:38.742432 22196 sys_catalog.cc:458] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:38.741992 22195 sys_catalog.cc:455] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f79eb86ba2fc46258d7ce18aea8ce88c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79eb86ba2fc46258d7ce18aea8ce88c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40013 } } }
I20260619 04:38:38.743043 22195 sys_catalog.cc:458] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:38.746367 22200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:38.746577 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:38.729972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51569 (local address 127.0.97.62:40013)
0619 04:38:38.730150 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:38.730157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:38.730653 (+ 496us) server_negotiation.cc:407] Connection header received
0619 04:38:38.730807 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:38.730810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:38.730869 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:38.730954 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:38.732749 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:38.733571 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:38.735252 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:38.735675 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:38.735759 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:38.738577 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:38.738584 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:38.738588 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:38.738625 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:38.742300 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:38.743301 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:38.743306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:38.743309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:38.743502 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:38.744040 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:38.744044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:38.744047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:38.744481 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:38.744793 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:38.746026 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0619 04:38:38.746201 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 04:38:38.751827 22200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:38.760821 22200 catalog_manager.cc:1347] Generated new cluster ID: e6651fa4e15e4c34a800c839b091ac96
I20260619 04:38:38.760952 22200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:38.774735 22200 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:38.776245 22200 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:38.783532 22200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f79eb86ba2fc46258d7ce18aea8ce88c: Generated new TSK 0
I20260619 04:38:38.784422 22200 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:38:38.794507 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 22117
I20260619 04:38:38.796259 22179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:38.885978 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 22117
W20260619 04:38:38.899542 22102 connection.cc:535] client connection to 127.0.97.62:40013 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:40013): Connection reset by peer (error 104)
2026-06-19T04:38:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1851 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:38:38 dist-test-slave-n3tq krb5kdc[22216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:38:38 dist-test-slave-n3tq krb5kdc[22216](info): set up 2 sockets
Jun 19 04:38:38 dist-test-slave-n3tq krb5kdc[22216](info): commencing operation
krb5kdc: starting...
W20260619 04:38:39.900580 388 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:38:40 dist-test-slave-n3tq krb5kdc[22216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:38:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:38:40.135326 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45517
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:40.456612 22229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:40.456892 22229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:40.457016 22229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:40.464251 22229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:40.464367 22229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:40.464432 22229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:40.464533 22229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:40.464593 22229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:40.474570 22229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45517
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:40.475867 22229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:40.477209 22229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:40.488488 22234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:40.488600 22235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:40.488615 22237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:40.489405 22229 server_base.cc:1034] running on GCE node
I20260619 04:38:40.490195 22229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:40.491592 22229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:40.492789 22229 hybrid_clock.cc:648] HybridClock initialized: now 1781843920492743 us; error 31 us; skew 500 ppm
I20260619 04:38:40.493125 22229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:40 dist-test-slave-n3tq krb5kdc[22216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:40.496068 22229 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:38:40.497489 22229 webserver.cc:460] Webserver started at http://127.0.97.62:44243/ using document root <none> and password file <none>
I20260619 04:38:40.498167 22229 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:40.498281 22229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:40.498577 22229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:40.502856 22229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "805e80ac8f334e85b40d70eaa61626cc"
format_stamp: "Formatted at 2026-06-19 04:38:40 on dist-test-slave-n3tq"
server_key: "e063eff802799e379227742c4329c595"
server_key_iv: "51a2924219b35f9b8ea3def1f1b7b475"
server_key_version: "encryptionkey@0"
I20260619 04:38:40.503854 22229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "805e80ac8f334e85b40d70eaa61626cc"
format_stamp: "Formatted at 2026-06-19 04:38:40 on dist-test-slave-n3tq"
server_key: "e063eff802799e379227742c4329c595"
server_key_iv: "51a2924219b35f9b8ea3def1f1b7b475"
server_key_version: "encryptionkey@0"
I20260619 04:38:40.509732 22229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 04:38:40.513036 22244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:40.514309 22229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:38:40.514470 22229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "805e80ac8f334e85b40d70eaa61626cc"
format_stamp: "Formatted at 2026-06-19 04:38:40 on dist-test-slave-n3tq"
server_key: "e063eff802799e379227742c4329c595"
server_key_iv: "51a2924219b35f9b8ea3def1f1b7b475"
server_key_version: "encryptionkey@0"
I20260619 04:38:40.514650 22229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:40.545871 22229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:40.550235 22229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:40.550498 22229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:40.567781 22229 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45517
I20260619 04:38:40.567826 22295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45517 every 8 connection(s)
I20260619 04:38:40.569140 22229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:40.572817 22296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:40.577678 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 22229
I20260619 04:38:40.577867 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:40.578415 388 external_mini_cluster.cc:1428] Setting key ca49c5d22853b41db80d5e066903efbf
I20260619 04:38:40.583974 22296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc: Bootstrap starting.
I20260619 04:38:40.587551 22296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:40.588825 22296 log.cc:824] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:40.591629 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:40.580152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57085 (local address 127.0.97.62:45517)
0619 04:38:40.580946 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:40.580959 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:40.581002 (+ 43us) server_negotiation.cc:407] Connection header received
0619 04:38:40.582691 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:40.582714 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:40.583118 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:40.583425 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:40.583963 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:40.585281 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:40.587048 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:40.587769 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:40.587898 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:40.589163 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:40.589170 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:40.589184 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:40.589924 (+ 740us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:40.590023 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:40.590055 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:40.590202 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:40.590514 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 04:38:40.590688 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":380,"threads_started":1}
I20260619 04:38:40.591887 22296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc: No bootstrap required, opened a new log
I20260619 04:38:40.595323 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:38:40.601244 22296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805e80ac8f334e85b40d70eaa61626cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45517 } }
I20260619 04:38:40.601686 22296 raft_consensus.cc:374] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:40.601812 22296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 805e80ac8f334e85b40d70eaa61626cc, State: Initialized, Role: FOLLOWER
I20260619 04:38:40.602480 22296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805e80ac8f334e85b40d70eaa61626cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45517 } }
I20260619 04:38:40.602758 22296 raft_consensus.cc:388] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:38:40.602906 22296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:38:40.603063 22296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:40.605885 22296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805e80ac8f334e85b40d70eaa61626cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45517 } }
I20260619 04:38:40.606529 22296 leader_election.cc:302] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 805e80ac8f334e85b40d70eaa61626cc; no voters:
I20260619 04:38:40.607506 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:40.600257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48511 (local address 127.0.97.62:45517)
0619 04:38:40.600443 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:40.600474 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:40.600903 (+ 429us) server_negotiation.cc:407] Connection header received
0619 04:38:40.601053 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:40.601058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:40.601138 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:40.601247 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:40.601652 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:40.602740 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:40.604102 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:40.604845 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:40.604969 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:40.606415 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:40.606422 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:40.606426 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:40.606448 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:40.606523 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:40.606574 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:40.606782 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:40.606899 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 04:38:40.606982 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 04:38:40.608433 22296 leader_election.cc:288] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:38:40.608577 22306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:40.612969 22296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:40.615456 22306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc [term 1 LEADER]: Becoming Leader. State: Replica: 805e80ac8f334e85b40d70eaa61626cc, State: Running, Role: LEADER
I20260619 04:38:40.615929 22306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805e80ac8f334e85b40d70eaa61626cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45517 } }
I20260619 04:38:40.618961 22308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 805e80ac8f334e85b40d70eaa61626cc. Latest consensus state: current_term: 1 leader_uuid: "805e80ac8f334e85b40d70eaa61626cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805e80ac8f334e85b40d70eaa61626cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45517 } } }
I20260619 04:38:40.618855 22307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "805e80ac8f334e85b40d70eaa61626cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805e80ac8f334e85b40d70eaa61626cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45517 } } }
I20260619 04:38:40.619270 22307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:40.619272 22308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc [sys.catalog]: This master's current role is: LEADER
W20260619 04:38:40.623850 22320 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 04:38:40.623955 22320 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 04:38:40.624058 22321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:40.625578 22321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:40.633352 22321 catalog_manager.cc:1347] Generated new cluster ID: 884d22a85d2a41c283d1a0ecc5162c22
I20260619 04:38:40.633453 22321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:40.648744 22321 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:40.650537 22321 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:40.664685 22321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 805e80ac8f334e85b40d70eaa61626cc: Generated new TSK 0
I20260619 04:38:40.665750 22321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:38:40.673348 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 22229
W20260619 04:38:40.674880 22291 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57085
W20260619 04:38:40.675554 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 22229: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:38:40.675700 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 22229
W20260619 04:38:40.689257 22214 connection.cc:535] client connection to 127.0.97.62:45517 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:45517): Connection reset by peer (error 104)
2026-06-19T04:38:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1789 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:38:40 dist-test-slave-n3tq krb5kdc[22327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:38:40 dist-test-slave-n3tq krb5kdc[22327](info): set up 2 sockets
Jun 19 04:38:40 dist-test-slave-n3tq krb5kdc[22327](info): commencing operation
krb5kdc: starting...
W20260619 04:38:41.676522 388 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:38:41 dist-test-slave-n3tq krb5kdc[22327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:38:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:38:41.909210 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36703
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:42.226922 22340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:42.227200 22340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:42.227330 22340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:42.237092 22340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:42.237210 22340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:42.237280 22340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:42.237363 22340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:42.237423 22340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:42.247359 22340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36703
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:42.248673 22340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:42.249939 22340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:42.261196 22345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:42.261222 22346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:42.261866 22348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:42.262187 22340 server_base.cc:1034] running on GCE node
I20260619 04:38:42.262880 22340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:42.264166 22340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:42.265357 22340 hybrid_clock.cc:648] HybridClock initialized: now 1781843922265312 us; error 33 us; skew 500 ppm
I20260619 04:38:42.265666 22340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:42 dist-test-slave-n3tq krb5kdc[22327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:42.274214 22340 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:38:42.275736 22340 webserver.cc:460] Webserver started at http://127.0.97.62:40213/ using document root <none> and password file <none>
I20260619 04:38:42.276408 22340 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:42.276546 22340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:42.276840 22340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:42.279829 22340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "1be1c912d56748a0bb664ed795af910a"
format_stamp: "Formatted at 2026-06-19 04:38:42 on dist-test-slave-n3tq"
server_key: "ce9c608be755462252ce6260879a69f6"
server_key_iv: "d21fe146809014cc30eca71ccae30071"
server_key_version: "encryptionkey@0"
I20260619 04:38:42.280633 22340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "1be1c912d56748a0bb664ed795af910a"
format_stamp: "Formatted at 2026-06-19 04:38:42 on dist-test-slave-n3tq"
server_key: "ce9c608be755462252ce6260879a69f6"
server_key_iv: "d21fe146809014cc30eca71ccae30071"
server_key_version: "encryptionkey@0"
I20260619 04:38:42.285866 22340 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 04:38:42.289448 22355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:42.291030 22340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:38:42.291244 22340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "1be1c912d56748a0bb664ed795af910a"
format_stamp: "Formatted at 2026-06-19 04:38:42 on dist-test-slave-n3tq"
server_key: "ce9c608be755462252ce6260879a69f6"
server_key_iv: "d21fe146809014cc30eca71ccae30071"
server_key_version: "encryptionkey@0"
I20260619 04:38:42.291443 22340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:42.297287 22340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:42.301249 22340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:42.301519 22340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:42.318535 22340 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36703
I20260619 04:38:42.318580 22406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36703 every 8 connection(s)
I20260619 04:38:42.319939 22340 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:42.322110 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 22340
I20260619 04:38:42.322283 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:42.322811 388 external_mini_cluster.cc:1428] Setting key e4b64aa1cd7f6c0878e4484aadb043dc
I20260619 04:38:42.324302 22407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:42.330245 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:42.324557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40239 (local address 127.0.97.62:36703)
0619 04:38:42.325320 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:42.325331 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:42.325372 (+ 41us) server_negotiation.cc:407] Connection header received
0619 04:38:42.326897 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:42.326919 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:42.327260 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:42.327585 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:42.328669 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:42.328683 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:42.328697 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:42.328711 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:42.328792 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:42.328816 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:42.328981 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:42.329079 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 04:38:42.329262 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":333,"threads_started":1}
I20260619 04:38:42.333748 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:38:42.336192 22407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a: Bootstrap starting.
I20260619 04:38:42.339949 22407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:42.340052 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:42.337908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58009 (local address 127.0.97.62:36703)
0619 04:38:42.338088 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:42.338093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:42.338346 (+ 253us) server_negotiation.cc:407] Connection header received
0619 04:38:42.338517 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:42.338521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:42.338567 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:42.338632 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:42.339495 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:42.339499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:42.339502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:42.339516 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:42.339550 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:42.339564 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:42.339642 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:42.339757 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 04:38:42.339800 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 04:38:42.341136 22407 log.cc:824] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:42.343705 22407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a: No bootstrap required, opened a new log
I20260619 04:38:42.350698 22407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be1c912d56748a0bb664ed795af910a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36703 } }
I20260619 04:38:42.351083 22407 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:42.351171 22407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be1c912d56748a0bb664ed795af910a, State: Initialized, Role: FOLLOWER
I20260619 04:38:42.351740 22407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be1c912d56748a0bb664ed795af910a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36703 } }
I20260619 04:38:42.351949 22407 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:38:42.352079 22407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:38:42.352229 22407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:42.354357 22407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be1c912d56748a0bb664ed795af910a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36703 } }
I20260619 04:38:42.354851 22407 leader_election.cc:302] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1be1c912d56748a0bb664ed795af910a; no voters:
I20260619 04:38:42.355571 22407 leader_election.cc:288] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:38:42.355685 22417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:42.356333 22417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a [term 1 LEADER]: Becoming Leader. State: Replica: 1be1c912d56748a0bb664ed795af910a, State: Running, Role: LEADER
I20260619 04:38:42.356869 22417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be1c912d56748a0bb664ed795af910a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36703 } }
I20260619 04:38:42.357559 22407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:42.359400 22418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1be1c912d56748a0bb664ed795af910a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be1c912d56748a0bb664ed795af910a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36703 } } }
I20260619 04:38:42.359421 22419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1be1c912d56748a0bb664ed795af910a. Latest consensus state: current_term: 1 leader_uuid: "1be1c912d56748a0bb664ed795af910a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be1c912d56748a0bb664ed795af910a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36703 } } }
I20260619 04:38:42.359905 22419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:42.359905 22418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:42.369488 22424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:42.373764 22424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:42.381564 22424 catalog_manager.cc:1347] Generated new cluster ID: 9c78abe4b9bf4032b6d2e2dd709ad0da
I20260619 04:38:42.381680 22424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:42.411639 22424 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:42.412125 22424 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:42.423477 22424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1be1c912d56748a0bb664ed795af910a: Generated new TSK 0
I20260619 04:38:42.424301 22424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:38:42.428534 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 22340
W20260619 04:38:42.429570 22402 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40239
W20260619 04:38:42.430289 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 22340: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:38:42.430423 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 22340
2026-06-19T04:38:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1754 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:38:42 dist-test-slave-n3tq krb5kdc[22438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:38:42 dist-test-slave-n3tq krb5kdc[22438](info): set up 2 sockets
Jun 19 04:38:42 dist-test-slave-n3tq krb5kdc[22438](info): commencing operation
krb5kdc: starting...
W20260619 04:38:43.436620 388 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:38:43 dist-test-slave-n3tq krb5kdc[22438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:38:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:38:43.667382 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33315
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:43.983968 22451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:43.984264 22451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:43.984406 22451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:43.991465 22451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:43.991586 22451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:43.991667 22451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:43.991747 22451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:43.991804 22451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:44.001681 22451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:44.002941 22451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:44.004211 22451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:44.015475 22456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:44.015448 22459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:44.015499 22457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:44.016745 22451 server_base.cc:1034] running on GCE node
I20260619 04:38:44.017551 22451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:44.019070 22451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:44.020262 22451 hybrid_clock.cc:648] HybridClock initialized: now 1781843924020222 us; error 50 us; skew 500 ppm
I20260619 04:38:44.020656 22451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:44 dist-test-slave-n3tq krb5kdc[22438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:44.023680 22451 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:38:44.025305 22451 webserver.cc:460] Webserver started at http://127.0.97.62:33361/ using document root <none> and password file <none>
I20260619 04:38:44.026002 22451 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:44.026118 22451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:44.026407 22451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:44.029485 22451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "3053cc5c82304168b6adc51d0e7dfd6b"
format_stamp: "Formatted at 2026-06-19 04:38:44 on dist-test-slave-n3tq"
server_key: "679f9998366bc0885513b52a83a4a09f"
server_key_iv: "230a5da2f660849212c70091838702f5"
server_key_version: "encryptionkey@0"
I20260619 04:38:44.030217 22451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "3053cc5c82304168b6adc51d0e7dfd6b"
format_stamp: "Formatted at 2026-06-19 04:38:44 on dist-test-slave-n3tq"
server_key: "679f9998366bc0885513b52a83a4a09f"
server_key_iv: "230a5da2f660849212c70091838702f5"
server_key_version: "encryptionkey@0"
I20260619 04:38:44.035307 22451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:38:44.038741 22466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:44.040199 22451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 04:38:44.040383 22451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "3053cc5c82304168b6adc51d0e7dfd6b"
format_stamp: "Formatted at 2026-06-19 04:38:44 on dist-test-slave-n3tq"
server_key: "679f9998366bc0885513b52a83a4a09f"
server_key_iv: "230a5da2f660849212c70091838702f5"
server_key_version: "encryptionkey@0"
I20260619 04:38:44.040583 22451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:44.065487 22451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:44.069540 22451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:44.069808 22451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:44.087417 22451 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33315
I20260619 04:38:44.087446 22517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33315 every 8 connection(s)
I20260619 04:38:44.088798 22451 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:44.090090 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 22451
I20260619 04:38:44.090269 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:44.090935 388 external_mini_cluster.cc:1428] Setting key 4db5b3b21c41eaa27f399f00a98e8ab5
I20260619 04:38:44.093642 22518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 04:38:44 dist-test-slave-n3tq krb5kdc[22438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:38:44.105121 22518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b: Bootstrap starting.
I20260619 04:38:44.108551 22518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:44.109593 22518 log.cc:824] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:44.112967 22518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b: No bootstrap required, opened a new log
I20260619 04:38:44.113346 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:44.092905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45649 (local address 127.0.97.62:33315)
0619 04:38:44.093655 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:44.093668 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:44.093717 (+ 49us) server_negotiation.cc:407] Connection header received
0619 04:38:44.095384 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:44.095407 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:44.095837 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:44.096235 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:44.097716 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:44.099096 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:44.100394 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:44.101129 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:44.101295 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:44.105690 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:44.105699 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:44.105715 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:44.105759 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:44.108813 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:44.109526 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:44.109532 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:44.109539 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:44.109720 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:44.110155 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:44.110159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:44.110163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:44.110975 (+ 812us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:44.111233 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:44.111654 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 04:38:44.112046 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":325,"threads_started":1}
I20260619 04:38:44.117033 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:38:44.122213 22518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3053cc5c82304168b6adc51d0e7dfd6b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33315 } }
I20260619 04:38:44.122712 22518 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:44.122830 22518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3053cc5c82304168b6adc51d0e7dfd6b, State: Initialized, Role: FOLLOWER
I20260619 04:38:44.123515 22518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3053cc5c82304168b6adc51d0e7dfd6b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33315 } }
I20260619 04:38:44.123799 22518 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:38:44.123945 22518 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:38:44.124119 22518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:44.126986 22518 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3053cc5c82304168b6adc51d0e7dfd6b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33315 } }
I20260619 04:38:44.127578 22518 leader_election.cc:302] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3053cc5c82304168b6adc51d0e7dfd6b; no voters:
I20260619 04:38:44.128149 22518 leader_election.cc:288] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:38:44.128235 22528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:44.129408 22528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b [term 1 LEADER]: Becoming Leader. State: Replica: 3053cc5c82304168b6adc51d0e7dfd6b, State: Running, Role: LEADER
I20260619 04:38:44.129922 22528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3053cc5c82304168b6adc51d0e7dfd6b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33315 } }
I20260619 04:38:44.130465 22518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:44.132488 22530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3053cc5c82304168b6adc51d0e7dfd6b. Latest consensus state: current_term: 1 leader_uuid: "3053cc5c82304168b6adc51d0e7dfd6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3053cc5c82304168b6adc51d0e7dfd6b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33315 } } }
I20260619 04:38:44.132825 22529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3053cc5c82304168b6adc51d0e7dfd6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3053cc5c82304168b6adc51d0e7dfd6b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33315 } } }
I20260619 04:38:44.132836 22530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:44.133131 22529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:44.136209 22534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:44.138113 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:44.122109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55009 (local address 127.0.97.62:33315)
0619 04:38:44.122316 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:44.122322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:44.122687 (+ 365us) server_negotiation.cc:407] Connection header received
0619 04:38:44.122860 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:44.122864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:44.122939 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:44.123025 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:44.124831 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:44.125613 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:44.127305 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:44.127813 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:44.127906 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:44.130789 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:44.130796 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:44.130801 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:44.130837 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:44.134790 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:44.135629 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:44.135634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:44.135638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:44.135802 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:44.136181 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:44.136186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:44.136194 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:44.136589 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:44.137595 (+ 1006us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:44.137703 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 04:38:44.137862 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 04:38:44.142040 22534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:44.151974 22534 catalog_manager.cc:1347] Generated new cluster ID: 8186fcb298254e4c9e58e76a0b9284e9
I20260619 04:38:44.152088 22534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:44.191905 22534 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:44.193234 22534 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:44.213785 22534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3053cc5c82304168b6adc51d0e7dfd6b: Generated new TSK 0
I20260619 04:38:44.214655 22534 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:38:44.219820 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 22451
I20260619 04:38:44.221350 22513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:44.314980 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 22451
2026-06-19T04:38:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1884 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:38:44 dist-test-slave-n3tq krb5kdc[22550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:38:44 dist-test-slave-n3tq krb5kdc[22550](info): set up 2 sockets
Jun 19 04:38:44 dist-test-slave-n3tq krb5kdc[22550](info): commencing operation
krb5kdc: starting...
W20260619 04:38:45.316617 388 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:38:45 dist-test-slave-n3tq krb5kdc[22550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:38:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:38:45.545877 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37465
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:45.861253 22563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:45.861531 22563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:45.861631 22563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:45.868677 22563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:45.868779 22563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:45.868831 22563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:45.868891 22563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:45.868932 22563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:45.878552 22563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:45.879748 22563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:45.880970 22563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:45.891808 22569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:45.891757 22571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:45.891839 22568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:45.892560 22563 server_base.cc:1034] running on GCE node
I20260619 04:38:45.893314 22563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:45.894639 22563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:45.895843 22563 hybrid_clock.cc:648] HybridClock initialized: now 1781843925895799 us; error 33 us; skew 500 ppm
I20260619 04:38:45.896164 22563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:45 dist-test-slave-n3tq krb5kdc[22550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:45.899091 22563 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:38:45.900420 22563 webserver.cc:460] Webserver started at http://127.0.97.62:37387/ using document root <none> and password file <none>
I20260619 04:38:45.901088 22563 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:45.901185 22563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:45.901508 22563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:45.905187 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "447e6238bdb847f68406de04d4803848"
format_stamp: "Formatted at 2026-06-19 04:38:45 on dist-test-slave-n3tq"
server_key: "4a09723dfc343b1a4e9da6398f3f6817"
server_key_iv: "28bf1db60d1d9b581a3127ff5f37813c"
server_key_version: "encryptionkey@0"
I20260619 04:38:45.905911 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "447e6238bdb847f68406de04d4803848"
format_stamp: "Formatted at 2026-06-19 04:38:45 on dist-test-slave-n3tq"
server_key: "4a09723dfc343b1a4e9da6398f3f6817"
server_key_iv: "28bf1db60d1d9b581a3127ff5f37813c"
server_key_version: "encryptionkey@0"
I20260619 04:38:45.911000 22563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:38:45.914408 22578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:45.915697 22563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 04:38:45.915850 22563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "447e6238bdb847f68406de04d4803848"
format_stamp: "Formatted at 2026-06-19 04:38:45 on dist-test-slave-n3tq"
server_key: "4a09723dfc343b1a4e9da6398f3f6817"
server_key_iv: "28bf1db60d1d9b581a3127ff5f37813c"
server_key_version: "encryptionkey@0"
I20260619 04:38:45.916023 22563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:45.927486 22563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:45.931589 22563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:45.931838 22563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:45.948554 22563 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37465
I20260619 04:38:45.948598 22629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37465 every 8 connection(s)
I20260619 04:38:45.949923 22563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:45.953509 22630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:45.958491 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 22563
I20260619 04:38:45.958648 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:45.959030 388 external_mini_cluster.cc:1428] Setting key 60235817d61e113064b78c13a515423d
I20260619 04:38:45.965250 22630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848: Bootstrap starting.
I20260619 04:38:45.968852 22630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:45.970132 22630 log.cc:824] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:45.973042 22630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848: No bootstrap required, opened a new log
I20260619 04:38:45.973346 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:45.960772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52685 (local address 127.0.97.62:37465)
0619 04:38:45.961618 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:45.961634 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:45.961684 (+ 50us) server_negotiation.cc:407] Connection header received
0619 04:38:45.964008 (+ 2324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:45.964036 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:45.964587 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:45.965008 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:45.965529 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:45.967464 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:45.968808 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:45.969811 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:45.969996 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:45.971116 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:45.971124 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:45.971138 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:45.971665 (+ 527us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:45.971735 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:45.971772 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:45.971867 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:45.972092 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 04:38:45.972344 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":339,"threads_started":1}
I20260619 04:38:45.977051 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:38:45.980841 22630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447e6238bdb847f68406de04d4803848" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37465 } }
I20260619 04:38:45.981319 22630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:45.981408 22630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 447e6238bdb847f68406de04d4803848, State: Initialized, Role: FOLLOWER
I20260619 04:38:45.982055 22630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447e6238bdb847f68406de04d4803848" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37465 } }
I20260619 04:38:45.982312 22630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:38:45.982450 22630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:38:45.982641 22630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:45.985280 22630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447e6238bdb847f68406de04d4803848" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37465 } }
I20260619 04:38:45.985842 22630 leader_election.cc:302] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 447e6238bdb847f68406de04d4803848; no voters:
I20260619 04:38:45.986464 22630 leader_election.cc:288] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:38:45.986601 22640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:45.987604 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:45.981078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45841 (local address 127.0.97.62:37465)
0619 04:38:45.981247 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:45.981262 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:45.981649 (+ 387us) server_negotiation.cc:407] Connection header received
0619 04:38:45.981848 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:45.981853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:45.981930 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:45.982020 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:45.982489 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:45.983518 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:45.984808 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:45.985427 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:45.985543 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:45.986579 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:45.986585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:45.986593 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:45.986614 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:45.986657 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:45.986678 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:45.986773 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:45.987212 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 04:38:45.987307 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 04:38:45.987919 22640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848 [term 1 LEADER]: Becoming Leader. State: Replica: 447e6238bdb847f68406de04d4803848, State: Running, Role: LEADER
I20260619 04:38:45.988672 22630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:45.988562 22640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447e6238bdb847f68406de04d4803848" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37465 } }
I20260619 04:38:45.991564 22642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 447e6238bdb847f68406de04d4803848. Latest consensus state: current_term: 1 leader_uuid: "447e6238bdb847f68406de04d4803848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447e6238bdb847f68406de04d4803848" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37465 } } }
I20260619 04:38:45.991595 22641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "447e6238bdb847f68406de04d4803848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447e6238bdb847f68406de04d4803848" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37465 } } }
I20260619 04:38:45.992000 22642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:45.992062 22641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:45.994331 22650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:45.998524 22650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:46.006495 22650 catalog_manager.cc:1347] Generated new cluster ID: 6fd715ab4f124813a501899e18ddd8b4
I20260619 04:38:46.006654 22650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:46.019102 22650 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:46.020488 22650 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:46.035943 22650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 447e6238bdb847f68406de04d4803848: Generated new TSK 0
I20260619 04:38:46.036895 22650 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:38:46.052472 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 22563
W20260619 04:38:46.053593 22625 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52685
W20260619 04:38:46.054278 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 22563: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:38:46.054425 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 22563
2026-06-19T04:38:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1739 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:38:46 dist-test-slave-n3tq krb5kdc[22661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:38:46 dist-test-slave-n3tq krb5kdc[22661](info): set up 2 sockets
Jun 19 04:38:46 dist-test-slave-n3tq krb5kdc[22661](info): commencing operation
krb5kdc: starting...
W20260619 04:38:47.076668 388 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:38:47 dist-test-slave-n3tq krb5kdc[22661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:38:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:38:47.313369 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37769
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:47.627386 22674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:47.627668 22674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:47.627769 22674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:47.634812 22674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:47.634913 22674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:47.634967 22674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:47.635025 22674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:47.635066 22674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:47.644435 22674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:47.645570 22674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:47.646759 22674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:47.657446 22679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:47.657471 22680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:47.657964 22682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:47.658200 22674 server_base.cc:1034] running on GCE node
I20260619 04:38:47.658922 22674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:47.660279 22674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:47.661502 22674 hybrid_clock.cc:648] HybridClock initialized: now 1781843927661430 us; error 62 us; skew 500 ppm
I20260619 04:38:47.661875 22674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:47 dist-test-slave-n3tq krb5kdc[22661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:47.668820 22674 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:38:47.670199 22674 webserver.cc:460] Webserver started at http://127.0.97.62:37563/ using document root <none> and password file <none>
I20260619 04:38:47.670833 22674 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:47.670933 22674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:47.671192 22674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:47.674057 22674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "9b653f7600674ddeb024f47545605f00"
format_stamp: "Formatted at 2026-06-19 04:38:47 on dist-test-slave-n3tq"
server_key: "6cf78ff3b3a4dc1f3dc17b28007dc43a"
server_key_iv: "e71e5d9089105fd892138136d33ff414"
server_key_version: "encryptionkey@0"
I20260619 04:38:47.674768 22674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "9b653f7600674ddeb024f47545605f00"
format_stamp: "Formatted at 2026-06-19 04:38:47 on dist-test-slave-n3tq"
server_key: "6cf78ff3b3a4dc1f3dc17b28007dc43a"
server_key_iv: "e71e5d9089105fd892138136d33ff414"
server_key_version: "encryptionkey@0"
I20260619 04:38:47.679654 22674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:38:47.682924 22689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:47.684173 22674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:38:47.684314 22674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "9b653f7600674ddeb024f47545605f00"
format_stamp: "Formatted at 2026-06-19 04:38:47 on dist-test-slave-n3tq"
server_key: "6cf78ff3b3a4dc1f3dc17b28007dc43a"
server_key_iv: "e71e5d9089105fd892138136d33ff414"
server_key_version: "encryptionkey@0"
I20260619 04:38:47.684561 22674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:47.689924 22674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:47.693503 22674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:47.693763 22674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:47.710430 22674 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37769
I20260619 04:38:47.710465 22740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37769 every 8 connection(s)
I20260619 04:38:47.711772 22674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:47.715325 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 22674
I20260619 04:38:47.715445 22741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:47.715497 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:47.716027 388 external_mini_cluster.cc:1428] Setting key 46dda5d9998ef63517eb51022a57ee10
I20260619 04:38:47.723531 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:47.717735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51483 (local address 127.0.97.62:37769)
0619 04:38:47.718535 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:47.718550 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:47.718588 (+ 38us) server_negotiation.cc:407] Connection header received
0619 04:38:47.720228 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:47.720251 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:47.720632 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:47.720975 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:47.721994 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:47.722001 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:47.722014 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:47.722040 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:47.722111 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:47.722131 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:47.722313 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:47.722421 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 04:38:47.722600 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":355,"threads_started":1}
I20260619 04:38:47.726986 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:38:47.728403 22741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00: Bootstrap starting.
I20260619 04:38:47.732050 22741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:47.733210 22741 log.cc:824] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:47.734117 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:47.731838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35573 (local address 127.0.97.62:37769)
0619 04:38:47.732023 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:47.732029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:47.732398 (+ 369us) server_negotiation.cc:407] Connection header received
0619 04:38:47.732539 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:47.732544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:47.732603 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:47.732712 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:47.733494 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:47.733499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:47.733503 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:38:47.733521 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:47.733565 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:38:47.733585 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:47.733720 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:47.733818 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 04:38:47.733868 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:38:47.735883 22741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00: No bootstrap required, opened a new log
I20260619 04:38:47.742660 22741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b653f7600674ddeb024f47545605f00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37769 } }
I20260619 04:38:47.742995 22741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:38:47.743085 22741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b653f7600674ddeb024f47545605f00, State: Initialized, Role: FOLLOWER
I20260619 04:38:47.743633 22741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b653f7600674ddeb024f47545605f00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37769 } }
I20260619 04:38:47.743840 22741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:38:47.743959 22741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:38:47.744089 22741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:47.746192 22741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b653f7600674ddeb024f47545605f00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37769 } }
I20260619 04:38:47.746640 22741 leader_election.cc:302] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b653f7600674ddeb024f47545605f00; no voters:
I20260619 04:38:47.747145 22741 leader_election.cc:288] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:38:47.747264 22751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:47.747833 22751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00 [term 1 LEADER]: Becoming Leader. State: Replica: 9b653f7600674ddeb024f47545605f00, State: Running, Role: LEADER
I20260619 04:38:47.748337 22751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b653f7600674ddeb024f47545605f00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37769 } }
I20260619 04:38:47.749086 22741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:47.751143 22752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b653f7600674ddeb024f47545605f00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b653f7600674ddeb024f47545605f00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37769 } } }
I20260619 04:38:47.751552 22752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:47.751596 22753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b653f7600674ddeb024f47545605f00. Latest consensus state: current_term: 1 leader_uuid: "9b653f7600674ddeb024f47545605f00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b653f7600674ddeb024f47545605f00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37769 } } }
I20260619 04:38:47.752137 22753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:47.758009 22760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:47.762756 22760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:47.770699 22760 catalog_manager.cc:1347] Generated new cluster ID: 059da53943884edeb835d3a7770836f8
I20260619 04:38:47.770805 22760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:47.787467 22760 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:47.787955 22760 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:47.795790 22760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b653f7600674ddeb024f47545605f00: Generated new TSK 0
I20260619 04:38:47.796630 22760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:38:47.807370 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 22674
W20260619 04:38:47.808396 22736 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51483
W20260619 04:38:47.809044 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 22674: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:38:47.809175 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 22674
2026-06-19T04:38:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1755 ms)
[----------] 18 tests from AuthTokenIssuingTest (32398 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:38:47 dist-test-slave-n3tq krb5kdc[22772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:38:47 dist-test-slave-n3tq krb5kdc[22772](info): set up 2 sockets
Jun 19 04:38:47 dist-test-slave-n3tq krb5kdc[22772](info): commencing operation
krb5kdc: starting...
W20260619 04:38:48.820760 388 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:38:48 dist-test-slave-n3tq krb5kdc[22772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:38:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:38:49.049453 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39281
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39281,127.0.97.61:41727,127.0.97.60:41007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:49.368132 22785 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 04:38:49.368499 22785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:49.368585 22785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:49.368667 22785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:49.375974 22785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:49.376075 22785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:49.376128 22785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:49.376176 22785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:49.376236 22785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:49.386174 22785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39281,127.0.97.61:41727,127.0.97.60:41007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:49.387419 22785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:49.388715 22785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:49.400027 22790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:49.399981 22793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:49.400094 22791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:49.400705 22785 server_base.cc:1034] running on GCE node
I20260619 04:38:49.401468 22785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:49.402815 22785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:49.404026 22785 hybrid_clock.cc:648] HybridClock initialized: now 1781843929403951 us; error 55 us; skew 500 ppm
I20260619 04:38:49.404333 22785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:49 dist-test-slave-n3tq krb5kdc[22772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:49.407258 22785 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:38:49.408689 22785 webserver.cc:460] Webserver started at http://127.0.97.62:44727/ using document root <none> and password file <none>
I20260619 04:38:49.409343 22785 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:49.409440 22785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:49.409742 22785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:49.412726 22785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "a6bf634089b94303ab9bf25fb4b3d681"
format_stamp: "Formatted at 2026-06-19 04:38:49 on dist-test-slave-n3tq"
server_key: "ca130c0691f46b93506f3cae7b1bc8de"
server_key_iv: "e1ffe8d7da259dd4b831ddbcee70e454"
server_key_version: "encryptionkey@0"
I20260619 04:38:49.413481 22785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "a6bf634089b94303ab9bf25fb4b3d681"
format_stamp: "Formatted at 2026-06-19 04:38:49 on dist-test-slave-n3tq"
server_key: "ca130c0691f46b93506f3cae7b1bc8de"
server_key_iv: "e1ffe8d7da259dd4b831ddbcee70e454"
server_key_version: "encryptionkey@0"
I20260619 04:38:49.418449 22785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:38:49.421802 22800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:49.423035 22785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:38:49.423182 22785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "a6bf634089b94303ab9bf25fb4b3d681"
format_stamp: "Formatted at 2026-06-19 04:38:49 on dist-test-slave-n3tq"
server_key: "ca130c0691f46b93506f3cae7b1bc8de"
server_key_iv: "e1ffe8d7da259dd4b831ddbcee70e454"
server_key_version: "encryptionkey@0"
I20260619 04:38:49.423369 22785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:49.447324 22785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:49.451282 22785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:49.451534 22785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:49.468899 22785 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39281
I20260619 04:38:49.468920 22851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39281 every 8 connection(s)
I20260619 04:38:49.470222 22785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:49.471411 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 22785
I20260619 04:38:49.471621 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:49.472384 388 external_mini_cluster.cc:1428] Setting key e039262cbbde41b97a4516845131e2f4
I20260619 04:38:49.474956 22852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:49.483596 22852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39281 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Jun 19 04:38:49 dist-test-slave-n3tq krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:38:49.507352 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:49.487016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39281 (local address 127.0.0.1:37146)
0619 04:38:49.487799 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:49.487843 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:49.489860 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:49.490694 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:49.490701 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:49.491391 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:49.491750 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:49.491768 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:49.493330 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:49.493334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:49.494413 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:49.494425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:49.495319 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:49.495326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:49.495579 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:49.496172 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:49.496198 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:49.500085 (+ 3887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:49.503115 (+ 3030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:49.503134 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:49.503147 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:49.503510 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:49.503880 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:49.503884 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:49.503888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:49.504034 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:49.504842 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:49.504850 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:49.505113 (+ 263us) client_negotiation.cc:769] Sending connection context
0619 04:38:49.505390 (+ 277us) client_negotiation.cc:240] Negotiation successful
0619 04:38:49.506026 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":160,"thread_start_us":326,"threads_started":1}
I20260619 04:38:49.507370 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:49.488340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37146 (local address 127.0.97.62:39281)
0619 04:38:49.489036 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:49.489044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:49.489092 (+ 48us) server_negotiation.cc:407] Connection header received
0619 04:38:49.490238 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:49.490243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:49.490477 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:49.490666 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:49.491939 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:49.493192 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:49.494541 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:49.495099 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:49.495271 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:49.500313 (+ 5042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:49.500322 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:49.500334 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:49.500367 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:49.502996 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:49.503624 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:49.503637 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:49.503639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:49.503779 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:49.504128 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:49.504132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:49.504135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:49.504716 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:49.504938 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:49.505525 (+ 587us) server_negotiation.cc:299] Negotiation successful
0619 04:38:49.505902 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":500,"thread_start_us":324,"threads_started":1}
I20260619 04:38:49.510594 22852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } has no permanent_uuid. Determining permanent_uuid...
I20260619 04:38:49.511672 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:49.511295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41727 (local address 127.0.0.1:53616)
0619 04:38:49.511447 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:49.511547 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 04:38:49.511943 22803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41727: connect: Connection refused (error 111)
I20260619 04:38:49.513433 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:49.513104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41727 (local address 127.0.0.1:53628)
0619 04:38:49.513247 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:49.513310 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 04:38:49.513603 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41727: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41727: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260619 04:38:49.525624 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41727
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39281,127.0.97.61:41727,127.0.97.60:41007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:38:49.562913 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } attempt: 1
I20260619 04:38:49.563971 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:49.563552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41727 (local address 127.0.0.1:53632)
0619 04:38:49.563735 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:49.563840 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 04:38:49.565099 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:49.564778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41727 (local address 127.0.0.1:53648)
0619 04:38:49.564902 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:49.564970 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 04:38:49.565302 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41727: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41727: connect: Connection refused (error 111)
I20260619 04:38:49.633549 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } attempt: 2
I20260619 04:38:49.634755 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:49.634250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41727 (local address 127.0.0.1:53664)
0619 04:38:49.634517 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:49.634610 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 04:38:49.635761 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:49.635417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41727 (local address 127.0.0.1:53676)
0619 04:38:49.635557 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:49.635637 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 04:38:49.635946 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41727: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41727: connect: Connection refused (error 111)
I20260619 04:38:49.727203 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } attempt: 3
I20260619 04:38:49.728363 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:49.727901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41727 (local address 127.0.0.1:53682)
0619 04:38:49.728130 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:49.728223 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 04:38:49.729444 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:49.729060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41727 (local address 127.0.0.1:53688)
0619 04:38:49.729219 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:49.729303 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 04:38:49.729651 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41727: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41727: connect: Connection refused (error 111)
W20260619 04:38:49.844379 22859 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 04:38:49.844726 22859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:49.844795 22859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:49.844877 22859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:49.851939 22859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:49.852034 22859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:49.852089 22859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:49.852138 22859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:49.852195 22859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:49.861920 22859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39281,127.0.97.61:41727,127.0.97.60:41007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:49.863144 22859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:49.864377 22859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 04:38:49.872859 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } attempt: 4
I20260619 04:38:49.873979 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:49.873455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41727 (local address 127.0.0.1:53698)
0619 04:38:49.873747 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:49.873834 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260619 04:38:49.875005 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:49.874682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41727 (local address 127.0.0.1:53702)
0619 04:38:49.874810 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:49.874890 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 04:38:49.875306 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41727: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41727: connect: Connection refused (error 111)
W20260619 04:38:49.876322 22864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:49.876302 22865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:49.876871 22867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:49.876899 22859 server_base.cc:1034] running on GCE node
I20260619 04:38:49.877717 22859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:49.879065 22859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:49.880275 22859 hybrid_clock.cc:648] HybridClock initialized: now 1781843929880209 us; error 42 us; skew 500 ppm
I20260619 04:38:49.880630 22859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:49 dist-test-slave-n3tq krb5kdc[22772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:49.883460 22859 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260619 04:38:49.884832 22859 webserver.cc:460] Webserver started at http://127.0.97.61:39945/ using document root <none> and password file <none>
I20260619 04:38:49.885468 22859 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:49.885560 22859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:49.885849 22859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:49.888775 22859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-1/data/instance:
uuid: "e1ae7c21b8804bdabd740d1578940c6c"
format_stamp: "Formatted at 2026-06-19 04:38:49 on dist-test-slave-n3tq"
server_key: "095221d4dc49d107d631f9cec0fab2c8"
server_key_iv: "a30c9c48b85ce11bc8d003a28696512d"
server_key_version: "encryptionkey@0"
I20260619 04:38:49.889499 22859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-1/wal/instance:
uuid: "e1ae7c21b8804bdabd740d1578940c6c"
format_stamp: "Formatted at 2026-06-19 04:38:49 on dist-test-slave-n3tq"
server_key: "095221d4dc49d107d631f9cec0fab2c8"
server_key_iv: "a30c9c48b85ce11bc8d003a28696512d"
server_key_version: "encryptionkey@0"
I20260619 04:38:49.894224 22859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 04:38:49.897447 22874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:49.898723 22859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:38:49.898876 22859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-1/wal
uuid: "e1ae7c21b8804bdabd740d1578940c6c"
format_stamp: "Formatted at 2026-06-19 04:38:49 on dist-test-slave-n3tq"
server_key: "095221d4dc49d107d631f9cec0fab2c8"
server_key_iv: "a30c9c48b85ce11bc8d003a28696512d"
server_key_version: "encryptionkey@0"
I20260619 04:38:49.899070 22859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:49.918889 22859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:49.922730 22859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:49.922971 22859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:49.938735 22859 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:41727
I20260619 04:38:49.938748 22925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:41727 every 8 connection(s)
I20260619 04:38:49.940024 22859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-1/data/info.pb
I20260619 04:38:49.943861 22926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:49.948690 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 22859
I20260619 04:38:49.948844 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-1/wal/instance
I20260619 04:38:49.949256 388 external_mini_cluster.cc:1428] Setting key 23780bfef663fb2dfc1bd3e4ead098e2
I20260619 04:38:49.950332 22926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39281 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:38:49 dist-test-slave-n3tq krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:38:49.969732 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:49.952930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37158 (local address 127.0.97.62:39281)
0619 04:38:49.953109 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:49.953114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:49.954435 (+ 1321us) server_negotiation.cc:407] Connection header received
0619 04:38:49.956325 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:49.956331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:49.956411 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:49.956518 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:49.958035 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:49.958740 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:49.960232 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:49.960698 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:49.960815 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:49.964200 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:49.964205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:49.964209 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:49.964242 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:49.966787 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:49.967454 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:49.967459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:49.967461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:49.967583 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:49.967976 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:49.967981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:49.967985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:49.968265 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:49.968389 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:49.969137 (+ 748us) server_negotiation.cc:299] Negotiation successful
0619 04:38:49.969392 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260619 04:38:49.970702 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:49.953529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39281 (local address 127.0.0.1:37158)
0619 04:38:49.954277 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:49.954314 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:49.956046 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:49.956759 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:49.956767 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:49.957267 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:49.957814 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:49.957831 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:49.958892 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:49.958896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:49.960082 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:49.960094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:49.960915 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:49.960922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:49.961147 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:49.961787 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:49.961807 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:49.963986 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:49.966918 (+ 2932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:49.966927 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:49.966944 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:49.967341 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:49.967692 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:49.967697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:49.967702 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:49.967874 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:49.968395 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:49.968403 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:49.968780 (+ 377us) client_negotiation.cc:769] Sending connection context
0619 04:38:49.969024 (+ 244us) client_negotiation.cc:240] Negotiation successful
0619 04:38:49.969532 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":306,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260619 04:38:49.972368 22926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:38:49 dist-test-slave-n3tq krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 04:38:49.986639 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:49.973116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41727 (local address 127.0.0.1:53712)
0619 04:38:49.973299 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:49.973317 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:49.973397 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:49.974280 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:49.974284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:49.974813 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:49.974954 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:49.974963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:49.976359 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:49.976363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:49.977227 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:49.977238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:49.978014 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:49.978020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:49.978168 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:49.978679 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:49.978695 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:49.980805 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:49.984382 (+ 3577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:49.984386 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:49.984389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:49.984813 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:49.985184 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:49.985189 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:49.985191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:49.985316 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:49.985952 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:49.985956 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:49.986082 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 04:38:49.986191 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 04:38:49.986380 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 04:38:49.987085 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:49.973312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53712 (local address 127.0.97.61:41727)
0619 04:38:49.973884 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:49.973891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:49.973916 (+ 25us) server_negotiation.cc:407] Connection header received
0619 04:38:49.973955 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:49.973961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:49.974148 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:49.974276 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:49.975139 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:49.976219 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:49.977349 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:49.977867 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:49.977974 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:49.981012 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:49.981025 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:49.981043 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:49.981083 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:49.984258 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:49.984930 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:49.984938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:49.984942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:49.985077 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:49.985411 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:49.985415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:49.985418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:49.985837 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:49.986028 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:49.986367 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 04:38:49.986610 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":310,"threads_started":1}
I20260619 04:38:49.989338 22926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } has no permanent_uuid. Determining permanent_uuid...
I20260619 04:38:49.990460 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:49.989976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41007 (local address 127.0.0.1:46736)
0619 04:38:49.990160 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:49.990290 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 04:38:49.990749 22875 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
I20260619 04:38:49.992308 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:49.991919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41007 (local address 127.0.0.1:46744)
0619 04:38:49.992083 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:49.992181 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 04:38:49.992573 22926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41007: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260619 04:38:50.003357 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41007
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39281,127.0.97.61:41727,127.0.97.60:41007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:38:50.041904 22926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } attempt: 1
I20260619 04:38:50.043098 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.042589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41007 (local address 127.0.0.1:46756)
0619 04:38:50.042792 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:50.042900 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 04:38:50.044071 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.043685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41007 (local address 127.0.0.1:46768)
0619 04:38:50.043834 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:50.043933 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 04:38:50.044268 22926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41007: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
I20260619 04:38:50.112514 22926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } attempt: 2
I20260619 04:38:50.113679 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.113171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41007 (local address 127.0.0.1:46784)
0619 04:38:50.113376 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:50.113494 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 04:38:50.114694 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.114300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41007 (local address 127.0.0.1:46788)
0619 04:38:50.114460 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:50.114546 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 04:38:50.114892 22926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41007: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
I20260619 04:38:50.174595 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } attempt: 5
Jun 19 04:38:50 dist-test-slave-n3tq krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 04:38:50.188153 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.175376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41727 (local address 127.0.0.1:53718)
0619 04:38:50.175585 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:50.175607 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:50.175728 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:50.176096 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:50.176101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:50.176569 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:50.176745 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:50.176755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.177730 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.177733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:50.178610 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:50.178623 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.179322 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.179327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:50.179516 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:50.180086 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:50.180104 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:50.182331 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:50.185951 (+ 3620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:50.185955 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:50.185958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:50.186326 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:50.186710 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:50.186713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:50.186716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:50.186838 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:50.187430 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:50.187433 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:50.187560 (+ 127us) client_negotiation.cc:769] Sending connection context
0619 04:38:50.187685 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 04:38:50.187898 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 04:38:50.188398 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.175397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53718 (local address 127.0.97.61:41727)
0619 04:38:50.175631 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:50.175637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:50.175677 (+ 40us) server_negotiation.cc:407] Connection header received
0619 04:38:50.175853 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:50.175858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:50.175935 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:50.176018 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:50.176892 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.177578 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.178754 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.179179 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.179287 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:50.182540 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:50.182548 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:50.182553 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:50.182594 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:50.185795 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.186447 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.186454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.186458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.186611 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.186930 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.186934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.186938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.187329 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:50.187488 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:50.187815 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 04:38:50.188068 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260619 04:38:50.189386 22852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } has no permanent_uuid. Determining permanent_uuid...
I20260619 04:38:50.190409 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.190048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41007 (local address 127.0.0.1:46790)
0619 04:38:50.190205 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:50.190277 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 04:38:50.191222 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.190929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41007 (local address 127.0.0.1:46802)
0619 04:38:50.191055 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:50.191130 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 04:38:50.191394 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41007: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
I20260619 04:38:50.206116 22926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } attempt: 3
I20260619 04:38:50.207376 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.206798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41007 (local address 127.0.0.1:46818)
0619 04:38:50.207050 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:50.207170 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 04:38:50.208429 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.207982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41007 (local address 127.0.0.1:46828)
0619 04:38:50.208205 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:50.208293 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260619 04:38:50.208679 22926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41007: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
I20260619 04:38:50.242617 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } attempt: 1
I20260619 04:38:50.243721 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.243257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41007 (local address 127.0.0.1:46832)
0619 04:38:50.243442 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:50.243526 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 04:38:50.244663 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.244317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41007 (local address 127.0.0.1:46836)
0619 04:38:50.244447 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:50.244552 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 04:38:50.244846 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41007: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
I20260619 04:38:50.313072 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } attempt: 2
I20260619 04:38:50.314317 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.313837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41007 (local address 127.0.0.1:46846)
0619 04:38:50.314061 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:50.314151 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 04:38:50.315227 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.314908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41007 (local address 127.0.0.1:46850)
0619 04:38:50.315051 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:50.315115 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 04:38:50.315436 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41007: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
W20260619 04:38:50.324227 22933 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 04:38:50.324651 22933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:50.324744 22933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:50.324860 22933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:50.332194 22933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:50.332297 22933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:50.332391 22933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:50.332477 22933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:50.332572 22933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:50.342455 22933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39281,127.0.97.61:41727,127.0.97.60:41007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:50.343716 22933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:50.345041 22933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 04:38:50.351881 22926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } attempt: 4
I20260619 04:38:50.352928 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.352443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41007 (local address 127.0.0.1:46860)
0619 04:38:50.352700 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:50.352784 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 04:38:50.353994 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.353649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41007 (local address 127.0.0.1:46868)
0619 04:38:50.353801 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:50.353874 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 04:38:50.354207 22926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41007: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
W20260619 04:38:50.356179 22938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:50.356235 22941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:50.356297 22939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:50.361819 22933 server_base.cc:1034] running on GCE node
I20260619 04:38:50.362649 22933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:50.364108 22933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:50.365322 22933 hybrid_clock.cc:648] HybridClock initialized: now 1781843930365255 us; error 42 us; skew 500 ppm
I20260619 04:38:50.365676 22933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:50 dist-test-slave-n3tq krb5kdc[22772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:50.368686 22933 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260619 04:38:50.370159 22933 webserver.cc:460] Webserver started at http://127.0.97.60:36903/ using document root <none> and password file <none>
I20260619 04:38:50.370849 22933 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:50.370954 22933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:50.371244 22933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:50.374181 22933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-2/data/instance:
uuid: "ccd736cdf47a458388fdf68b88eed6bb"
format_stamp: "Formatted at 2026-06-19 04:38:50 on dist-test-slave-n3tq"
server_key: "caffd44284c48daee2b5375be9c23bec"
server_key_iv: "62651f8b13ac468b8c9d6cd26ebe3cab"
server_key_version: "encryptionkey@0"
I20260619 04:38:50.374909 22933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-2/wal/instance:
uuid: "ccd736cdf47a458388fdf68b88eed6bb"
format_stamp: "Formatted at 2026-06-19 04:38:50 on dist-test-slave-n3tq"
server_key: "caffd44284c48daee2b5375be9c23bec"
server_key_iv: "62651f8b13ac468b8c9d6cd26ebe3cab"
server_key_version: "encryptionkey@0"
I20260619 04:38:50.379822 22933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:38:50.383270 22948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:50.384663 22933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:38:50.384822 22933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-2/wal
uuid: "ccd736cdf47a458388fdf68b88eed6bb"
format_stamp: "Formatted at 2026-06-19 04:38:50 on dist-test-slave-n3tq"
server_key: "caffd44284c48daee2b5375be9c23bec"
server_key_iv: "62651f8b13ac468b8c9d6cd26ebe3cab"
server_key_version: "encryptionkey@0"
I20260619 04:38:50.385026 22933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:50.405455 22933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:50.409416 22933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:50.409664 22933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:50.421634 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } attempt: 3
I20260619 04:38:50.422757 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.422277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41007 (local address 127.0.0.1:46874)
0619 04:38:50.422509 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:50.422594 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 04:38:50.423827 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.423415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41007 (local address 127.0.0.1:46886)
0619 04:38:50.423633 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:50.423722 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260619 04:38:50.424069 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41007: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41007: connect: Connection refused (error 111)
I20260619 04:38:50.427021 22933 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:41007
I20260619 04:38:50.427013 22999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:41007 every 8 connection(s)
I20260619 04:38:50.428366 22933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-2/data/info.pb
I20260619 04:38:50.431906 23000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:50.436578 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 22933
I20260619 04:38:50.436743 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781843588881932-388-0/minicluster-data/master-2/wal/instance
I20260619 04:38:50.437157 388 external_mini_cluster.cc:1428] Setting key e0d5fe68aeeea784c89f1d71c3e811c6
I20260619 04:38:50.438480 23000 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39281 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:38:50 dist-test-slave-n3tq krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 19 04:38:50 dist-test-slave-n3tq krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 04:38:50.457952 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.440968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37172 (local address 127.0.97.62:39281)
0619 04:38:50.441160 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:50.441166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:50.441914 (+ 748us) server_negotiation.cc:407] Connection header received
0619 04:38:50.442211 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:50.442217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:50.442290 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:50.442390 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:50.443996 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.445092 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.447094 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.447682 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.447806 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:50.451680 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:50.451685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:50.451688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:50.451724 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:50.454899 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.455630 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.455636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.455639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.455762 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.456210 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.456214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.456218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.456584 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:50.456704 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:50.457395 (+ 691us) server_negotiation.cc:299] Negotiation successful
0619 04:38:50.457592 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:38:50.458921 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.438902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46892 (local address 127.0.97.60:41007)
0619 04:38:50.439629 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:50.439641 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:50.439681 (+ 40us) server_negotiation.cc:407] Connection header received
0619 04:38:50.441577 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:50.441600 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:50.442005 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:50.442238 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:50.443917 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.445534 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.447258 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.448083 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.448168 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:50.452240 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:50.452252 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:50.452269 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:50.452308 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:50.455494 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.456206 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.456215 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.456218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.456345 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.456735 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.456740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.456744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.457094 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:50.457341 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:50.457690 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 04:38:50.457920 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":532,"spinlock_wait_cycles":20864,"thread_start_us":328,"threads_started":1}
I20260619 04:38:50.458937 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.441018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39281 (local address 127.0.0.1:37172)
0619 04:38:50.441767 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:50.441807 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:50.441978 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:50.442484 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:50.442491 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:50.443278 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:50.443801 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:50.443826 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.445258 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.445262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:50.446937 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:50.446966 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.447890 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.447895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:50.448115 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:50.448872 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:50.448909 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:50.451460 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:50.455022 (+ 3562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:50.455031 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:50.455041 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:50.455521 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:50.455860 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:50.455865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:50.455868 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:50.456114 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:50.456700 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:50.456708 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:50.457051 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 04:38:50.457288 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 04:38:50.457794 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":62,"thread_start_us":377,"threads_started":1}
I20260619 04:38:50.460391 23000 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:38:50 dist-test-slave-n3tq krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 19 04:38:50 dist-test-slave-n3tq krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:38:50.473188 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.461036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41727 (local address 127.0.0.1:53724)
0619 04:38:50.461189 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:50.461204 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:50.461285 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:50.461592 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:50.461596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:50.461941 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:50.462112 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:50.462124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.463046 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.463051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:50.464007 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:50.464018 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.464699 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.464705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:50.464895 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:50.465466 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:50.465487 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:50.467742 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:50.471072 (+ 3330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:50.471078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:50.471082 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:50.471441 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:50.471779 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:50.471784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:50.471787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:50.471915 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:50.472387 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:50.472392 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:50.472568 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 04:38:50.472717 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 04:38:50.472944 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 04:38:50.473397 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.461095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53724 (local address 127.0.97.61:41727)
0619 04:38:50.461308 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:50.461315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:50.461333 (+ 18us) server_negotiation.cc:407] Connection header received
0619 04:38:50.461390 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:50.461395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:50.461472 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:50.461551 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:50.462240 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.462902 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.464140 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.464578 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.464682 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:50.467913 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:50.467917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:50.467921 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:50.467959 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:50.470900 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.471548 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.471553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.471557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.471683 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.472018 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.472023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.472027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.472277 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:50.472409 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:50.472843 (+ 434us) server_negotiation.cc:299] Negotiation successful
0619 04:38:50.473053 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 04:38:50.474157 23000 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } has no permanent_uuid. Determining permanent_uuid...
I20260619 04:38:50.478998 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.461946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37188 (local address 127.0.97.62:39281)
0619 04:38:50.462113 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:50.462118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:50.462135 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:38:50.463076 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:50.463081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:50.463150 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:50.463242 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:50.464376 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.465189 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.466896 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.467342 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.467447 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:50.471898 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:50.471904 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:50.471909 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:50.471947 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:50.476164 (+ 4217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.476993 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.476999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.477002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.477154 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.477555 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.477559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.477562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.477873 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:50.478140 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:50.478448 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 04:38:50.478661 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 04:38:50 dist-test-slave-n3tq krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 04:38:50.487756 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.475142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46896 (local address 127.0.97.60:41007)
0619 04:38:50.475273 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:50.475278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:50.475292 (+ 14us) server_negotiation.cc:407] Connection header received
0619 04:38:50.475327 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:50.475330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:50.475383 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:50.475474 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:50.476092 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.477043 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.478332 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.478900 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.479027 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:50.482341 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:50.482346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:50.482349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:50.482378 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:50.485313 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.486036 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.486042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.486045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.486193 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.486590 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.486593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.486595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.486840 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:50.486941 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:50.487355 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 04:38:50.487517 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 04:38:50.488144 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.474934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41007 (local address 127.0.0.1:46896)
0619 04:38:50.475077 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:50.475094 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:50.475179 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:50.475494 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:50.475498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:50.475842 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:50.475976 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:50.475984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.477142 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.477148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:50.478194 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:50.478207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.479030 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.479034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:50.479261 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:50.479759 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:50.479775 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:50.482139 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:50.485448 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:50.485454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:50.485457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:50.485911 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:50.486314 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:50.486319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:50.486323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:50.486493 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:50.486961 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:50.486966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:50.487120 (+ 154us) client_negotiation.cc:769] Sending connection context
0619 04:38:50.487289 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 04:38:50.487700 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":92}
Jun 19 04:38:50 dist-test-slave-n3tq krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 04:38:50.493870 23000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccd736cdf47a458388fdf68b88eed6bb: Bootstrap starting.
I20260619 04:38:50.495833 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.480593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.0.97.61:41727)
0619 04:38:50.480742 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:50.480746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:50.480761 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:38:50.480900 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:50.480904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:50.480961 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:50.481030 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:50.482469 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.483485 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.484670 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.485314 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.485438 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:50.490159 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:50.490166 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:50.490171 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:50.490209 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:50.493285 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.493939 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.493944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.493947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.494100 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.494483 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.494487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.494490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.494816 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:50.495063 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:50.495246 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 04:38:50.495495 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 04:38:50.497680 23000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccd736cdf47a458388fdf68b88eed6bb: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:50.498770 23000 log.cc:824] T 00000000000000000000000000000000 P ccd736cdf47a458388fdf68b88eed6bb: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:50.500372 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:38:50.501464 23000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccd736cdf47a458388fdf68b88eed6bb: No bootstrap required, opened a new log
I20260619 04:38:50.502854 22827 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a6bf634089b94303ab9bf25fb4b3d681"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37188
I20260619 04:38:50.511526 23000 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccd736cdf47a458388fdf68b88eed6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf634089b94303ab9bf25fb4b3d681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39281 } } peers { permanent_uuid: "e1ae7c21b8804bdabd740d1578940c6c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } } peers { permanent_uuid: "ccd736cdf47a458388fdf68b88eed6bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } }
I20260619 04:38:50.511945 23000 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccd736cdf47a458388fdf68b88eed6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccd736cdf47a458388fdf68b88eed6bb, State: Initialized, Role: FOLLOWER
I20260619 04:38:50.512432 23000 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccd736cdf47a458388fdf68b88eed6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf634089b94303ab9bf25fb4b3d681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39281 } } peers { permanent_uuid: "e1ae7c21b8804bdabd740d1578940c6c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } } peers { permanent_uuid: "ccd736cdf47a458388fdf68b88eed6bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } }
I20260619 04:38:50.513342 23006 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd736cdf47a458388fdf68b88eed6bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf634089b94303ab9bf25fb4b3d681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39281 } } peers { permanent_uuid: "e1ae7c21b8804bdabd740d1578940c6c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } } peers { permanent_uuid: "ccd736cdf47a458388fdf68b88eed6bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } } }
I20260619 04:38:50.513672 23006 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd736cdf47a458388fdf68b88eed6bb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:38:50.514559 23000 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccd736cdf47a458388fdf68b88eed6bb [sys.catalog]: configured and running, proceeding with master startup.
W20260619 04:38:50.526226 23017 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ccd736cdf47a458388fdf68b88eed6bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 04:38:50.526358 23017 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 04:38:50.524039 22933 master_runner.cc:186] Error getting master registration for 127.0.97.62:39281: OK, instance_id { permanent_uuid: "a6bf634089b94303ab9bf25fb4b3d681" instance_seqno: 1781843929463929 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 04:38:50.535099 22933 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 04:38:50.601287 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } attempt: 4
Jun 19 04:38:50 dist-test-slave-n3tq krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 04:38:50.619000 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.602085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41007 (local address 127.0.0.1:46906)
0619 04:38:50.602307 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:50.602329 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:50.602466 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:50.602950 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:50.602955 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:50.603470 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:50.603689 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:50.603701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.605234 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.605241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:50.606732 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:50.606749 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.607925 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.607934 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:50.608252 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:50.609000 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:50.609025 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:50.611866 (+ 2841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:50.616193 (+ 4327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:50.616197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:50.616200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:50.616629 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:50.617053 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:50.617058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:50.617062 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:50.617248 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:50.617960 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:50.617965 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:50.618152 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 04:38:50.618296 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 04:38:50.618614 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 04:38:50.619004 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.602309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.0.97.60:41007)
0619 04:38:50.602549 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:50.602555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:50.602587 (+ 32us) server_negotiation.cc:407] Connection header received
0619 04:38:50.602654 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:50.602660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:50.602739 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:50.602842 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:50.603840 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.605032 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.606923 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.607716 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.607847 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:50.612100 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:50.612106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:50.612111 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:50.612154 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:50.616059 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.616742 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.616748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.616752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.616934 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.617348 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.617353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.617356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.617859 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:50.618001 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:50.618389 (+ 388us) server_negotiation.cc:299] Negotiation successful
0619 04:38:50.618619 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 04:38:50.624934 22852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681: Bootstrap starting.
I20260619 04:38:50.629031 22852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:50.630323 22852 log.cc:824] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:50.633634 22852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681: No bootstrap required, opened a new log
I20260619 04:38:50.641212 22852 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf634089b94303ab9bf25fb4b3d681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39281 } } peers { permanent_uuid: "e1ae7c21b8804bdabd740d1578940c6c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } } peers { permanent_uuid: "ccd736cdf47a458388fdf68b88eed6bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } }
I20260619 04:38:50.641669 22852 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6bf634089b94303ab9bf25fb4b3d681, State: Initialized, Role: FOLLOWER
I20260619 04:38:50.642211 22852 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf634089b94303ab9bf25fb4b3d681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39281 } } peers { permanent_uuid: "e1ae7c21b8804bdabd740d1578940c6c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } } peers { permanent_uuid: "ccd736cdf47a458388fdf68b88eed6bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } }
I20260619 04:38:50.643244 23020 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf634089b94303ab9bf25fb4b3d681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39281 } } peers { permanent_uuid: "e1ae7c21b8804bdabd740d1578940c6c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } } peers { permanent_uuid: "ccd736cdf47a458388fdf68b88eed6bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } } }
I20260619 04:38:50.643622 23020 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:38:50.644570 22852 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:50.644629 22827 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 04:38:50.644820 22827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:50.647280 22827 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf634089b94303ab9bf25fb4b3d681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39281 } } peers { permanent_uuid: "e1ae7c21b8804bdabd740d1578940c6c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } } peers { permanent_uuid: "ccd736cdf47a458388fdf68b88eed6bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } }
I20260619 04:38:50.649449 22975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a6bf634089b94303ab9bf25fb4b3d681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ccd736cdf47a458388fdf68b88eed6bb"
I20260619 04:38:50.649950 22975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccd736cdf47a458388fdf68b88eed6bb [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:50.653393 22975 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ccd736cdf47a458388fdf68b88eed6bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6bf634089b94303ab9bf25fb4b3d681 in term 1.
I20260619 04:38:50.653410 22926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } attempt: 5
I20260619 04:38:50.654091 22801 leader_election.cc:302] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6bf634089b94303ab9bf25fb4b3d681, ccd736cdf47a458388fdf68b88eed6bb; no voters:
I20260619 04:38:50.654635 22827 leader_election.cc:288] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681 [CANDIDATE]: Term 1 election: Requested vote from peers e1ae7c21b8804bdabd740d1578940c6c (127.0.97.61:41727), ccd736cdf47a458388fdf68b88eed6bb (127.0.97.60:41007)
I20260619 04:38:50.654975 23020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681 [term 1 FOLLOWER]: Leader election won for term 1
W20260619 04:38:50.661705 22954 tablet.cc:2369] T 00000000000000000000000000000000 P ccd736cdf47a458388fdf68b88eed6bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 04:38:50 dist-test-slave-n3tq krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 04:38:50.671525 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.654004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41007 (local address 127.0.0.1:46920)
0619 04:38:50.654433 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:50.654453 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:50.654581 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:50.655859 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:50.655864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:50.656366 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:50.656696 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:50.656709 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.658132 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.658138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:50.659546 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:50.659562 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.660750 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.660756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:50.660969 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:50.661686 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:50.661711 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:50.664488 (+ 2777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:50.668641 (+ 4153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:50.668646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:50.668650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:50.669195 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:50.669627 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:50.669631 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:50.669635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:50.669799 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:50.670482 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:50.670487 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:50.670649 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 04:38:50.670787 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 04:38:50.671095 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305}
I20260619 04:38:50.672549 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.654788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46920 (local address 127.0.97.60:41007)
0619 04:38:50.655540 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:50.655546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:50.655574 (+ 28us) server_negotiation.cc:407] Connection header received
0619 04:38:50.655627 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:50.655633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:50.655710 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:50.655796 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:50.656879 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.657924 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.659720 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.660411 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.660537 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:50.664703 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:50.664710 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:50.664716 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:50.664757 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:50.668429 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.669336 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.669342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.669346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.669505 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.669922 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.669927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.669930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.670284 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:50.670422 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:50.671916 (+ 1494us) server_negotiation.cc:299] Negotiation successful
0619 04:38:50.672174 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629}
I20260619 04:38:50.678632 22926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1ae7c21b8804bdabd740d1578940c6c: Bootstrap starting.
I20260619 04:38:50.684044 23020 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681 [term 1 LEADER]: Becoming Leader. State: Replica: a6bf634089b94303ab9bf25fb4b3d681, State: Running, Role: LEADER
I20260619 04:38:50.684122 22926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1ae7c21b8804bdabd740d1578940c6c: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:50.684883 23020 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf634089b94303ab9bf25fb4b3d681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39281 } } peers { permanent_uuid: "e1ae7c21b8804bdabd740d1578940c6c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } } peers { permanent_uuid: "ccd736cdf47a458388fdf68b88eed6bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } }
I20260619 04:38:50.686185 22926 log.cc:824] T 00000000000000000000000000000000 P e1ae7c21b8804bdabd740d1578940c6c: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:50.692153 23026 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6bf634089b94303ab9bf25fb4b3d681. Latest consensus state: current_term: 1 leader_uuid: "a6bf634089b94303ab9bf25fb4b3d681" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf634089b94303ab9bf25fb4b3d681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39281 } } peers { permanent_uuid: "e1ae7c21b8804bdabd740d1578940c6c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } } peers { permanent_uuid: "ccd736cdf47a458388fdf68b88eed6bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } } }
I20260619 04:38:50.692764 23026 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:50.698766 23034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:50.703676 22926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1ae7c21b8804bdabd740d1578940c6c: No bootstrap required, opened a new log
W20260619 04:38:50.704303 22806 tablet.cc:2369] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:38:50.712347 23034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:50.728776 22926 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1ae7c21b8804bdabd740d1578940c6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf634089b94303ab9bf25fb4b3d681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39281 } } peers { permanent_uuid: "e1ae7c21b8804bdabd740d1578940c6c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } } peers { permanent_uuid: "ccd736cdf47a458388fdf68b88eed6bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } }
I20260619 04:38:50.729295 22926 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1ae7c21b8804bdabd740d1578940c6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1ae7c21b8804bdabd740d1578940c6c, State: Initialized, Role: FOLLOWER
I20260619 04:38:50.729871 22926 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1ae7c21b8804bdabd740d1578940c6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf634089b94303ab9bf25fb4b3d681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39281 } } peers { permanent_uuid: "e1ae7c21b8804bdabd740d1578940c6c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } } peers { permanent_uuid: "ccd736cdf47a458388fdf68b88eed6bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } }
I20260619 04:38:50.748639 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.703852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46928 (local address 127.0.97.60:41007)
0619 04:38:50.704037 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:50.704043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:50.710382 (+ 6339us) server_negotiation.cc:407] Connection header received
0619 04:38:50.711221 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:50.711228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:50.711346 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:50.711422 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:50.718968 (+ 7546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.720077 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.727030 (+ 6953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.727716 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.727823 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:50.737437 (+ 9614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:50.737446 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:50.737450 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:50.737497 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:50.743846 (+ 6349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.745164 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.745169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.745173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.745344 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.746048 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.746053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.746069 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.746449 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:50.746705 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:50.747637 (+ 932us) server_negotiation.cc:299] Negotiation successful
0619 04:38:50.748018 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 04:38:50.751605 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.704549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37196 (local address 127.0.97.62:39281)
0619 04:38:50.704715 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:50.704721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:50.715692 (+ 10971us) server_negotiation.cc:407] Connection header received
0619 04:38:50.716135 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:50.716141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:50.716226 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:50.716298 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:50.726503 (+ 10205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.727600 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.732930 (+ 5330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.733684 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.733787 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:50.744484 (+ 10697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:50.744492 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:50.744497 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:50.744544 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:50.748195 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.749279 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.749284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.749287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.749467 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.749922 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.749927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.749930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.750286 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:50.750525 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:50.751034 (+ 509us) server_negotiation.cc:299] Negotiation successful
0619 04:38:50.751277 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 04:38:50.760253 22975 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ccd736cdf47a458388fdf68b88eed6bb [term 1 FOLLOWER]: Refusing update from remote peer a6bf634089b94303ab9bf25fb4b3d681: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:38:50.760725 23041 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ae7c21b8804bdabd740d1578940c6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf634089b94303ab9bf25fb4b3d681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39281 } } peers { permanent_uuid: "e1ae7c21b8804bdabd740d1578940c6c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } } peers { permanent_uuid: "ccd736cdf47a458388fdf68b88eed6bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } } }
I20260619 04:38:50.761181 23041 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ae7c21b8804bdabd740d1578940c6c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:38:50.761601 23026 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccd736cdf47a458388fdf68b88eed6bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:38:50.763056 22926 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1ae7c21b8804bdabd740d1578940c6c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:50.763142 22901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a6bf634089b94303ab9bf25fb4b3d681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e1ae7c21b8804bdabd740d1578940c6c"
I20260619 04:38:50.763602 22901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1ae7c21b8804bdabd740d1578940c6c [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:50.767129 22901 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e1ae7c21b8804bdabd740d1578940c6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6bf634089b94303ab9bf25fb4b3d681 in term 1.
I20260619 04:38:50.769954 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:50.703381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53750 (local address 127.0.97.61:41727)
0619 04:38:50.703555 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:50.703561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:50.713046 (+ 9485us) server_negotiation.cc:407] Connection header received
0619 04:38:50.713297 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:50.713302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:50.713383 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:50.713453 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:50.717625 (+ 4172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.718717 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.725180 (+ 6463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:50.725854 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:50.725971 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:50.738573 (+ 12602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:50.738581 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:50.738585 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:50.738637 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:50.742250 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.752506 (+ 10256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.752512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.752515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.752692 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:50.764655 (+ 11963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:50.764662 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:50.764666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:50.765197 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:50.765556 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:50.769295 (+ 3739us) server_negotiation.cc:299] Negotiation successful
0619 04:38:50.769586 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:38:50.774976 22901 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e1ae7c21b8804bdabd740d1578940c6c [term 1 FOLLOWER]: Refusing update from remote peer a6bf634089b94303ab9bf25fb4b3d681: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:38:50.778743 23026 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1ae7c21b8804bdabd740d1578940c6c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:38:50.787395 23041 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ae7c21b8804bdabd740d1578940c6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6bf634089b94303ab9bf25fb4b3d681. Latest consensus state: current_term: 1 leader_uuid: "a6bf634089b94303ab9bf25fb4b3d681" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf634089b94303ab9bf25fb4b3d681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39281 } } peers { permanent_uuid: "e1ae7c21b8804bdabd740d1578940c6c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } } peers { permanent_uuid: "ccd736cdf47a458388fdf68b88eed6bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } } }
I20260619 04:38:50.787736 23041 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ae7c21b8804bdabd740d1578940c6c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:38:50.801046 23043 mvcc.cc:205] Tried to move back new op lower bound from 7298432740380418048 to 7298432740093009920. Current Snapshot: MvccSnapshot[applied={T|T < 7298432740380418048}]
I20260619 04:38:50.802675 23026 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a6bf634089b94303ab9bf25fb4b3d681" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf634089b94303ab9bf25fb4b3d681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39281 } } peers { permanent_uuid: "e1ae7c21b8804bdabd740d1578940c6c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } } peers { permanent_uuid: "ccd736cdf47a458388fdf68b88eed6bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } } }
I20260619 04:38:50.803013 23026 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:50.805032 23041 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ae7c21b8804bdabd740d1578940c6c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a6bf634089b94303ab9bf25fb4b3d681" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf634089b94303ab9bf25fb4b3d681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39281 } } peers { permanent_uuid: "e1ae7c21b8804bdabd740d1578940c6c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } } peers { permanent_uuid: "ccd736cdf47a458388fdf68b88eed6bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } } }
I20260619 04:38:50.805317 23041 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ae7c21b8804bdabd740d1578940c6c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:38:50.806413 23046 mvcc.cc:205] Tried to move back new op lower bound from 7298432740380418048 to 7298432740093009920. Current Snapshot: MvccSnapshot[applied={T|T < 7298432740380418048}]
I20260619 04:38:50.808624 23006 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd736cdf47a458388fdf68b88eed6bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6bf634089b94303ab9bf25fb4b3d681. Latest consensus state: current_term: 1 leader_uuid: "a6bf634089b94303ab9bf25fb4b3d681" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf634089b94303ab9bf25fb4b3d681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39281 } } peers { permanent_uuid: "e1ae7c21b8804bdabd740d1578940c6c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } } peers { permanent_uuid: "ccd736cdf47a458388fdf68b88eed6bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } } }
I20260619 04:38:50.808969 23006 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd736cdf47a458388fdf68b88eed6bb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:38:50.812836 23020 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a6bf634089b94303ab9bf25fb4b3d681" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf634089b94303ab9bf25fb4b3d681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39281 } } peers { permanent_uuid: "e1ae7c21b8804bdabd740d1578940c6c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } } peers { permanent_uuid: "ccd736cdf47a458388fdf68b88eed6bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } } }
I20260619 04:38:50.813207 23020 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:50.820567 23034 catalog_manager.cc:1347] Generated new cluster ID: 9ba84563d50640aa8adf8d981cd35f83
I20260619 04:38:50.820693 23034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:50.829171 23006 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd736cdf47a458388fdf68b88eed6bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a6bf634089b94303ab9bf25fb4b3d681" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf634089b94303ab9bf25fb4b3d681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39281 } } peers { permanent_uuid: "e1ae7c21b8804bdabd740d1578940c6c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41727 } } peers { permanent_uuid: "ccd736cdf47a458388fdf68b88eed6bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41007 } } }
I20260619 04:38:50.829466 23006 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd736cdf47a458388fdf68b88eed6bb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:38:50.846179 23034 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:50.851670 23034 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:50.842192 22785 master_runner.cc:186] Error getting master registration for 127.0.97.61:41727: OK, instance_id { permanent_uuid: "e1ae7c21b8804bdabd740d1578940c6c" instance_seqno: 1781843929934239 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 04:38:50.852389 22785 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 04:38:50.879624 23034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6bf634089b94303ab9bf25fb4b3d681: Generated new TSK 0
I20260619 04:38:50.880923 23034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:38:50.885547 23063 catalog_manager.cc:1259] Loaded cluster ID: 9ba84563d50640aa8adf8d981cd35f83
I20260619 04:38:50.885648 23063 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e1ae7c21b8804bdabd740d1578940c6c: loading cluster ID for follower catalog manager: success
I20260619 04:38:50.888984 23063 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e1ae7c21b8804bdabd740d1578940c6c: acquiring CA information for follower catalog manager: success
I20260619 04:38:50.891252 23063 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e1ae7c21b8804bdabd740d1578940c6c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 04:38:51.036417 22930 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:51.016051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.0.97.61:41727)
0619 04:38:51.016259 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.016266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.033162 (+ 16896us) server_negotiation.cc:407] Connection header received
0619 04:38:51.034791 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.034798 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.034911 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.035002 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:51.036160 (+ 1158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53752: BlockingRecv error: recv got EOF from 127.0.0.1:53752 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 04:38:51.038909 22856 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:51.031286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37210 (local address 127.0.97.62:39281)
0619 04:38:51.031509 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.031516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.037334 (+ 5818us) server_negotiation.cc:407] Connection header received
0619 04:38:51.037549 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.037556 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.037640 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.037707 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:51.038663 (+ 956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37210: BlockingRecv error: recv got EOF from 127.0.0.1:37210 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 04:38:51.039999 23003 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:51.018064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46936 (local address 127.0.97.60:41007)
0619 04:38:51.018260 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.018267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.037225 (+ 18958us) server_negotiation.cc:407] Connection header received
0619 04:38:51.037473 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.037479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.037574 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.037644 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:51.039785 (+ 2141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46936: BlockingRecv error: recv got EOF from 127.0.0.1:46936 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 04:38:51.087390 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:51.071760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53764 (local address 127.0.97.61:41727)
0619 04:38:51.071942 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.071949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.079520 (+ 7571us) server_negotiation.cc:407] Connection header received
0619 04:38:51.080074 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.080081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.080288 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.080434 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:51.081133 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.082295 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.084314 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.085023 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.085124 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:51.085864 (+ 740us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:51.086569 (+ 705us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:51.086691 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:51.086943 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 04:38:51.087054 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 04:38:51.112856 23003 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:51.110580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46952 (local address 127.0.97.60:41007)
0619 04:38:51.110797 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.110803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.111269 (+ 466us) server_negotiation.cc:407] Connection header received
0619 04:38:51.111510 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.111516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.111655 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.111766 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:51.112612 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46952: BlockingRecv error: recv got EOF from 127.0.0.1:46952 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 04:38:51.131033 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:51.122006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53772 (local address 127.0.97.61:41727)
0619 04:38:51.122205 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.122211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.124975 (+ 2764us) server_negotiation.cc:407] Connection header received
0619 04:38:51.125178 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.125184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.125297 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.125394 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:51.125846 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.127022 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.128981 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.129669 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.129796 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:51.130249 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:51.130410 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:51.130497 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:51.130639 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 04:38:51.130723 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 04:38:51.139864 23003 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:51.138354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46966 (local address 127.0.97.60:41007)
0619 04:38:51.138549 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.138555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.138574 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:38:51.138639 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.138644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.138722 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.138819 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:51.139643 (+ 824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46966: BlockingRecv error: recv got EOF from 127.0.0.1:46966 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:38:51.154630 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:51.146186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53774 (local address 127.0.97.61:41727)
0619 04:38:51.146359 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.146365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.148977 (+ 2612us) server_negotiation.cc:407] Connection header received
0619 04:38:51.149189 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.149194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.149294 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.149378 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:51.149783 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.150767 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.152625 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.153265 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.153381 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:51.153847 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:51.153994 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:51.154084 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:51.154215 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 04:38:51.154296 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 04:38:51.167513 23003 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:51.165083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46970 (local address 127.0.97.60:41007)
0619 04:38:51.165256 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.165261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.165792 (+ 531us) server_negotiation.cc:407] Connection header received
0619 04:38:51.165974 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.165979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.166103 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.166222 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:51.167278 (+ 1056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46970: BlockingRecv error: recv got EOF from 127.0.0.1:46970 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:38:51.181020 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:51.174279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53784 (local address 127.0.97.61:41727)
0619 04:38:51.174439 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.174445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.174985 (+ 540us) server_negotiation.cc:407] Connection header received
0619 04:38:51.175166 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.175172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.175264 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.175369 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:51.175820 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.177027 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.178870 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.179615 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.179745 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:51.180183 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:51.180315 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:51.180412 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:51.180615 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 04:38:51.180703 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 04:38:51.190388 23003 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:51.188024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46976 (local address 127.0.97.60:41007)
0619 04:38:51.188284 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.188290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.188823 (+ 533us) server_negotiation.cc:407] Connection header received
0619 04:38:51.189031 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.189037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.189161 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.189271 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:51.190159 (+ 888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46976: BlockingRecv error: recv got EOF from 127.0.0.1:46976 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
I20260619 04:38:51.209517 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:51.202431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53786 (local address 127.0.97.61:41727)
0619 04:38:51.202627 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.202633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.203189 (+ 556us) server_negotiation.cc:407] Connection header received
0619 04:38:51.203405 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.203411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.203507 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.203620 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:51.204088 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.205303 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.207237 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.208016 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.208166 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:51.208668 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:51.208846 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:51.208953 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:51.209098 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 04:38:51.209187 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 04:38:51.227811 23003 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:51.225431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46988 (local address 127.0.97.60:41007)
0619 04:38:51.225613 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.225619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.226172 (+ 553us) server_negotiation.cc:407] Connection header received
0619 04:38:51.226463 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.226469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.226608 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.226713 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:51.227592 (+ 879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46988: BlockingRecv error: recv got EOF from 127.0.0.1:46988 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:38:51.254562 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:51.247732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53796 (local address 127.0.97.61:41727)
0619 04:38:51.247908 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.247914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.248433 (+ 519us) server_negotiation.cc:407] Connection header received
0619 04:38:51.248676 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.248683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.248787 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.248887 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:51.249354 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.250570 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.252426 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.253197 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.253346 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:51.253804 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:51.253947 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:51.254041 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:51.254179 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 04:38:51.254279 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 04:38:51.264107 23003 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:51.262163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46992 (local address 127.0.97.60:41007)
0619 04:38:51.262344 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.262350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.262371 (+ 21us) server_negotiation.cc:407] Connection header received
0619 04:38:51.262570 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.262578 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.262680 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.262784 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:51.263808 (+ 1024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46992: BlockingRecv error: recv got EOF from 127.0.0.1:46992 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 04:38:51.305045 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:51.299235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.0.97.61:41727)
0619 04:38:51.299451 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.299457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.299660 (+ 203us) server_negotiation.cc:407] Connection header received
0619 04:38:51.299823 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.299829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.299943 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.300072 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:51.300404 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.301544 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.302912 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.303634 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.303793 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:51.304176 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:51.304310 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:51.304402 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:51.304614 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 04:38:51.304705 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260619 04:38:51.312825 23003 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:51.310945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.0.97.60:41007)
0619 04:38:51.311135 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.311140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.311444 (+ 304us) server_negotiation.cc:407] Connection header received
0619 04:38:51.311584 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.311588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.311659 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.311739 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:51.312587 (+ 848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47002: BlockingRecv error: recv got EOF from 127.0.0.1:47002 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 04:38:51.384876 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:51.380159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.0.97.61:41727)
0619 04:38:51.380319 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.380324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.380684 (+ 360us) server_negotiation.cc:407] Connection header received
0619 04:38:51.380846 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.380850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.380932 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.381014 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:51.381445 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.382131 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.383397 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.383785 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.383880 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:51.384256 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:51.384368 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:51.384443 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:51.384588 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 04:38:51.384655 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 04:38:51.393535 23003 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:51.391738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47018 (local address 127.0.97.60:41007)
0619 04:38:51.391969 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.391975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.392239 (+ 264us) server_negotiation.cc:407] Connection header received
0619 04:38:51.392438 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.392442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.392544 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.392628 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:51.393365 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47018: BlockingRecv error: recv got EOF from 127.0.0.1:47018 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 04:38:51.528558 23017 catalog_manager.cc:1259] Loaded cluster ID: 9ba84563d50640aa8adf8d981cd35f83
I20260619 04:38:51.528672 23017 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ccd736cdf47a458388fdf68b88eed6bb: loading cluster ID for follower catalog manager: success
I20260619 04:38:51.530105 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:51.524956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53830 (local address 127.0.97.61:41727)
0619 04:38:51.525195 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.525202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.525430 (+ 228us) server_negotiation.cc:407] Connection header received
0619 04:38:51.525591 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.525595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.525679 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.525759 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:51.526116 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.526972 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.528297 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.528917 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.529046 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:51.529440 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:51.529568 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:51.529675 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:51.529774 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 04:38:51.529849 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 04:38:51.531818 23017 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ccd736cdf47a458388fdf68b88eed6bb: acquiring CA information for follower catalog manager: success
I20260619 04:38:51.534214 23017 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ccd736cdf47a458388fdf68b88eed6bb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 04:38:51.545048 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:51.540013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47030 (local address 127.0.97.60:41007)
0619 04:38:51.540230 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.540236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.540497 (+ 261us) server_negotiation.cc:407] Connection header received
0619 04:38:51.540679 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.540683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.540802 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.540895 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:51.541201 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.541899 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.543091 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.543466 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.543554 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:51.543944 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:51.544500 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:51.544616 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:51.544721 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 04:38:51.544790 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:38:51.559042 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:51.551010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47044 (local address 127.0.97.60:41007)
0619 04:38:51.551191 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.551197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.552988 (+ 1791us) server_negotiation.cc:407] Connection header received
0619 04:38:51.553170 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.553175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.553269 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.553354 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:51.553847 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.554905 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.557039 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.557656 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.557776 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:51.558346 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:51.558438 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:51.558484 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:51.558687 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 04:38:51.558766 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 04:38:51.559777 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:51.551038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53846 (local address 127.0.97.61:41727)
0619 04:38:51.551196 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.551201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.552636 (+ 1435us) server_negotiation.cc:407] Connection header received
0619 04:38:51.552892 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.552904 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.553021 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.553108 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:51.554267 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.555384 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.557312 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.557730 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.557844 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:51.558324 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:51.558451 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:51.558535 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:51.559524 (+ 989us) server_negotiation.cc:299] Negotiation successful
0619 04:38:51.559589 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 04:38:51.560837 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:51.550621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37212 (local address 127.0.97.62:39281)
0619 04:38:51.551952 (+ 1331us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:51.551958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:51.552514 (+ 556us) server_negotiation.cc:407] Connection header received
0619 04:38:51.552660 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:51.552666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:51.553214 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:51.553375 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:51.554267 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.555385 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.558257 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:51.558837 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:51.558952 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:51.559577 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:51.560012 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:51.560118 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:51.560491 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 04:38:51.560578 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1220,"thread_start_us":1125,"threads_started":1}
I20260619 04:38:51.565766 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 22785
I20260619 04:38:51.566790 22847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:51.675884 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 22785
I20260619 04:38:51.692988 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 22859
I20260619 04:38:51.694018 22921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:51.799580 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 22859
I20260619 04:38:51.815809 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 22933
I20260619 04:38:51.816915 22995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:51.921682 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 22933
2026-06-19T04:38:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4115 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:38:52 dist-test-slave-n3tq krb5kdc[23178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:38:52 dist-test-slave-n3tq krb5kdc[23178](info): set up 2 sockets
Jun 19 04:38:52 dist-test-slave-n3tq krb5kdc[23178](info): commencing operation
krb5kdc: starting...
W20260619 04:38:52.948709 388 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.008s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/test-user.keytab.
Jun 19 04:38:53 dist-test-slave-n3tq krb5kdc[23178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:38:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:38:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 04:38:53.187834 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41635
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41635,127.0.97.61:42857,127.0.97.60:39523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:38:53.507160 23191 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 04:38:53.507484 23191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:53.507555 23191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:53.507635 23191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:53.515036 23191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:53.515138 23191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:53.515194 23191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:53.515244 23191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:53.515322 23191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:53.525617 23191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41635,127.0.97.61:42857,127.0.97.60:39523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:53.526872 23191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:53.528117 23191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:53.539137 23196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:53.539103 23197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:53.539064 23199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:53.539824 23191 server_base.cc:1034] running on GCE node
I20260619 04:38:53.540585 23191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:53.541908 23191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:53.543115 23191 hybrid_clock.cc:648] HybridClock initialized: now 1781843933543042 us; error 47 us; skew 500 ppm
I20260619 04:38:53.543426 23191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:53 dist-test-slave-n3tq krb5kdc[23178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:53.549290 23191 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 04:38:53.550731 23191 webserver.cc:460] Webserver started at http://127.0.97.62:38661/ using document root <none> and password file <none>
I20260619 04:38:53.551426 23191 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:53.551523 23191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:53.551810 23191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:53.554746 23191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-0/data/instance:
uuid: "46a5f9f6b6e5409989986cd1b6c56b85"
format_stamp: "Formatted at 2026-06-19 04:38:53 on dist-test-slave-n3tq"
server_key: "624bb20895e3ccad8f033e413bc576d3"
server_key_iv: "d1d1a4d18d9edfe3064b4c48021badc0"
server_key_version: "encryptionkey@0"
I20260619 04:38:53.555505 23191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-0/wal/instance:
uuid: "46a5f9f6b6e5409989986cd1b6c56b85"
format_stamp: "Formatted at 2026-06-19 04:38:53 on dist-test-slave-n3tq"
server_key: "624bb20895e3ccad8f033e413bc576d3"
server_key_iv: "d1d1a4d18d9edfe3064b4c48021badc0"
server_key_version: "encryptionkey@0"
I20260619 04:38:53.560483 23191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 04:38:53.563745 23206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:53.565083 23191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:38:53.565243 23191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-0/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-0/wal
uuid: "46a5f9f6b6e5409989986cd1b6c56b85"
format_stamp: "Formatted at 2026-06-19 04:38:53 on dist-test-slave-n3tq"
server_key: "624bb20895e3ccad8f033e413bc576d3"
server_key_iv: "d1d1a4d18d9edfe3064b4c48021badc0"
server_key_version: "encryptionkey@0"
I20260619 04:38:53.565438 23191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:53.580143 23191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:53.584290 23191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:53.584581 23191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:53.601229 23191 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41635
I20260619 04:38:53.601246 23257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41635 every 8 connection(s)
I20260619 04:38:53.602550 23191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-0/data/info.pb
I20260619 04:38:53.606294 23258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:53.611189 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 23191
I20260619 04:38:53.611366 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-0/wal/instance
I20260619 04:38:53.611869 388 external_mini_cluster.cc:1428] Setting key 48619822bfc9e687a529146b11ef5cf9
I20260619 04:38:53.613209 23258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41635 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:38:53 dist-test-slave-n3tq krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:38:53.636674 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:53.616801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41635 (local address 127.0.0.1:39968)
0619 04:38:53.617662 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:53.617707 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:53.619957 (+ 2250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:53.620847 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:53.620855 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:53.621578 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:53.621905 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:53.621922 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:53.624022 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:53.624026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:53.625139 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:53.625151 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:53.626245 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:53.626251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:53.626402 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:53.627015 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:53.627037 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:53.629126 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:53.632813 (+ 3687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:53.632822 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:53.632831 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:53.633270 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:53.633722 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:53.633725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:53.633727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:53.633876 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:53.634810 (+ 934us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:53.634816 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:53.635012 (+ 196us) client_negotiation.cc:769] Sending connection context
0619 04:38:53.635212 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 04:38:53.635616 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":370,"threads_started":1}
I20260619 04:38:53.636755 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:53.616801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39968 (local address 127.0.97.62:41635)
0619 04:38:53.618505 (+ 1704us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:53.618514 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:53.618555 (+ 41us) server_negotiation.cc:407] Connection header received
0619 04:38:53.620317 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:53.620322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:53.620621 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:53.620774 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:53.622107 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:53.623865 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:53.625266 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:53.626003 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:53.626156 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:53.629347 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:53.629356 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:53.629370 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:53.629401 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:53.632682 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:53.633415 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:53.633424 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:53.633427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:53.633615 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:53.633984 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:53.633988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:53.633992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:53.634668 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:53.634874 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:53.635409 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 04:38:53.635792 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":1512,"spinlock_wait_cycles":16128,"thread_start_us":1291,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260619 04:38:53.639394 23258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260619 04:38:53.640355 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:53.639999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42857 (local address 127.0.0.1:43132)
0619 04:38:53.640161 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:53.640257 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 04:38:53.640579 23210 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42857: connect: Connection refused (error 111)
I20260619 04:38:53.641952 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:53.641626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42857 (local address 127.0.0.1:43136)
0619 04:38:53.641762 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:53.641821 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 04:38:53.642128 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42857: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42857: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260619 04:38:53.665520 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42857
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41635,127.0.97.61:42857,127.0.97.60:39523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:38:53.691423 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } attempt: 1
I20260619 04:38:53.692484 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:53.692049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42857 (local address 127.0.0.1:43140)
0619 04:38:53.692229 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:53.692311 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 04:38:53.693507 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:53.693139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42857 (local address 127.0.0.1:43152)
0619 04:38:53.693282 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:53.693345 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 04:38:53.693692 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42857: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42857: connect: Connection refused (error 111)
I20260619 04:38:53.761911 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } attempt: 2
I20260619 04:38:53.763057 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:53.762630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42857 (local address 127.0.0.1:43168)
0619 04:38:53.762825 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:53.762915 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 04:38:53.763988 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:53.763665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42857 (local address 127.0.0.1:43176)
0619 04:38:53.763783 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:53.763861 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 04:38:53.764173 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42857: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42857: connect: Connection refused (error 111)
I20260619 04:38:53.855427 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } attempt: 3
I20260619 04:38:53.856544 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:53.856063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42857 (local address 127.0.0.1:43184)
0619 04:38:53.856258 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:53.856360 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 04:38:53.857476 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:53.857136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42857 (local address 127.0.0.1:43192)
0619 04:38:53.857271 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:53.857343 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 04:38:53.857684 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42857: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42857: connect: Connection refused (error 111)
W20260619 04:38:53.981125 23265 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 04:38:53.981467 23265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:53.981539 23265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:53.981619 23265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:53.988785 23265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:53.988878 23265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:53.988931 23265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:53.988979 23265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:53.989039 23265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:53.998879 23265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41635,127.0.97.61:42857,127.0.97.60:39523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:54.000137 23265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:54.000921 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } attempt: 4
I20260619 04:38:54.001557 23265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 04:38:54.002161 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.001624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42857 (local address 127.0.0.1:43198)
0619 04:38:54.001931 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.002035 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260619 04:38:54.003189 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.002841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42857 (local address 127.0.0.1:43200)
0619 04:38:54.002978 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.003062 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 04:38:54.003365 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42857: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42857: connect: Connection refused (error 111)
W20260619 04:38:54.012928 23271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:54.012889 23270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:54.012951 23273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:54.014010 23265 server_base.cc:1034] running on GCE node
I20260619 04:38:54.014770 23265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:54.016160 23265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:54.017379 23265 hybrid_clock.cc:648] HybridClock initialized: now 1781843934017303 us; error 39 us; skew 500 ppm
I20260619 04:38:54.017699 23265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:54 dist-test-slave-n3tq krb5kdc[23178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:54.020785 23265 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260619 04:38:54.022240 23265 webserver.cc:460] Webserver started at http://127.0.97.61:36281/ using document root <none> and password file <none>
I20260619 04:38:54.022912 23265 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:54.022996 23265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:54.023257 23265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:54.026243 23265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-1/data/instance:
uuid: "8d6c3bc032d2453dac5a40bad157a9d4"
format_stamp: "Formatted at 2026-06-19 04:38:54 on dist-test-slave-n3tq"
server_key: "a7fd4c6ca85479c007a6b4f58bb440bc"
server_key_iv: "811b9aa51e1d6749ef63029be3ba9101"
server_key_version: "encryptionkey@0"
I20260619 04:38:54.026991 23265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-1/wal/instance:
uuid: "8d6c3bc032d2453dac5a40bad157a9d4"
format_stamp: "Formatted at 2026-06-19 04:38:54 on dist-test-slave-n3tq"
server_key: "a7fd4c6ca85479c007a6b4f58bb440bc"
server_key_iv: "811b9aa51e1d6749ef63029be3ba9101"
server_key_version: "encryptionkey@0"
I20260619 04:38:54.031999 23265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260619 04:38:54.035388 23280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:54.036695 23265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:38:54.036839 23265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-1/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-1/wal
uuid: "8d6c3bc032d2453dac5a40bad157a9d4"
format_stamp: "Formatted at 2026-06-19 04:38:54 on dist-test-slave-n3tq"
server_key: "a7fd4c6ca85479c007a6b4f58bb440bc"
server_key_iv: "811b9aa51e1d6749ef63029be3ba9101"
server_key_version: "encryptionkey@0"
I20260619 04:38:54.037031 23265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:54.058032 23265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:54.062139 23265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:54.062354 23265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:54.079706 23265 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:42857
I20260619 04:38:54.079726 23331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:42857 every 8 connection(s)
I20260619 04:38:54.081069 23265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-1/data/info.pb
I20260619 04:38:54.084635 23332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:54.088498 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 23265
I20260619 04:38:54.088687 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-1/wal/instance
I20260619 04:38:54.089210 388 external_mini_cluster.cc:1428] Setting key 8dd76646827e53ea2d8c9edfa19e6a96
I20260619 04:38:54.092747 23332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41635 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:38:54 dist-test-slave-n3tq krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260619 04:38:54.114071 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.096032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39982 (local address 127.0.97.62:41635)
0619 04:38:54.096239 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:54.096245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:54.097637 (+ 1392us) server_negotiation.cc:407] Connection header received
0619 04:38:54.100072 (+ 2435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:54.100078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:54.100164 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:54.100254 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:54.101893 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.102572 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.104122 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.104601 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.104715 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:54.108183 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:54.108188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:54.108191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:54.108226 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:54.111000 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.111746 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.111751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.111753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.111875 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.112266 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.112271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.112274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.112610 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:54.112754 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:54.113490 (+ 736us) server_negotiation.cc:299] Negotiation successful
0619 04:38:54.113760 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260619 04:38:54.115180 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.096587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41635 (local address 127.0.0.1:39982)
0619 04:38:54.097433 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.097492 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:54.099751 (+ 2259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:54.100553 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:54.100562 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:54.101122 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:54.101693 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:54.101710 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.102730 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.102734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:54.103974 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:54.103986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.104832 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.104840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:54.105082 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:54.105770 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:54.105794 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:54.107951 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:54.111157 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:54.111167 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:54.111185 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:54.111626 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:54.111978 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:54.111983 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:54.111987 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:54.112158 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:54.112727 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:54.112736 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:54.113135 (+ 399us) client_negotiation.cc:769] Sending connection context
0619 04:38:54.113386 (+ 251us) client_negotiation.cc:240] Negotiation successful
0619 04:38:54.113939 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":336,"threads_started":1}
I20260619 04:38:54.116912 23332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:38:54 dist-test-slave-n3tq krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 04:38:54.130702 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.117717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42857 (local address 127.0.0.1:43210)
0619 04:38:54.117901 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.117919 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:54.118003 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:54.118850 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:54.118855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:54.119245 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:54.119397 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:54.119405 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.120763 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.120767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:54.121587 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:54.121597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.122356 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.122361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:54.122515 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:54.123106 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:54.123123 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:54.125365 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:54.128284 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:54.128290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:54.128295 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:54.128681 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:54.129021 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:54.129026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:54.129030 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:54.129157 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:54.129727 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:54.129732 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:54.129898 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 04:38:54.130011 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 04:38:54.130290 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 04:38:54.130762 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.117888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43210 (local address 127.0.97.61:42857)
0619 04:38:54.118462 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:54.118468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:54.118492 (+ 24us) server_negotiation.cc:407] Connection header received
0619 04:38:54.118533 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:54.118538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:54.118712 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:54.118846 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:54.119543 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.120636 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.121701 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.122220 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.122324 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:54.125563 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:54.125576 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:54.125590 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:54.125621 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:54.128151 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.128799 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.128807 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.128809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.128923 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.129256 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.129261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.129264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.129616 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:54.129815 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:54.130177 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 04:38:54.130394 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":302,"threads_started":1}
I20260619 04:38:54.133462 23332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } has no permanent_uuid. Determining permanent_uuid...
I20260619 04:38:54.134590 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.134099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39523 (local address 127.0.0.1:55704)
0619 04:38:54.134266 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.134408 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 04:38:54.134912 23284 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
I20260619 04:38:54.136421 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.136130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39523 (local address 127.0.0.1:55706)
0619 04:38:54.136244 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.136318 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 04:38:54.136718 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39523: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260619 04:38:54.143316 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iQRF/build/asan/bin/kudu
/tmp/dist-test-task24iQRF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39523
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41635,127.0.97.61:42857,127.0.97.60:39523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:38:54.186020 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } attempt: 1
I20260619 04:38:54.187192 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.186743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39523 (local address 127.0.0.1:55720)
0619 04:38:54.186947 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.187036 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 04:38:54.188123 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.187785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39523 (local address 127.0.0.1:55732)
0619 04:38:54.187948 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.188014 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 04:38:54.188315 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39523: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
I20260619 04:38:54.256544 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } attempt: 2
I20260619 04:38:54.257618 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.257154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39523 (local address 127.0.0.1:55736)
0619 04:38:54.257350 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.257439 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 04:38:54.258535 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.258217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39523 (local address 127.0.0.1:55740)
0619 04:38:54.258355 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.258423 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 04:38:54.258728 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39523: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
I20260619 04:38:54.302618 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } attempt: 5
Jun 19 04:38:54 dist-test-slave-n3tq krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 04:38:54.317207 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.303266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42857 (local address 127.0.0.1:43226)
0619 04:38:54.303511 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.303534 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:54.303669 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:54.304020 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:54.304025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:54.304577 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:54.304820 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:54.304834 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.305825 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.305830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:54.307181 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:54.307205 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.307981 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.307987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:54.308174 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:54.308814 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:54.308838 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:54.311144 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:54.314928 (+ 3784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:54.314932 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:54.314935 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:54.315323 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:54.315726 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:54.315730 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:54.315732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:54.315878 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:54.316485 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:54.316490 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:54.316631 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 04:38:54.316750 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 04:38:54.316959 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 04:38:54.317411 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.303539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43226 (local address 127.0.97.61:42857)
0619 04:38:54.303699 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:54.303703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:54.303744 (+ 41us) server_negotiation.cc:407] Connection header received
0619 04:38:54.303799 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:54.303802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:54.303877 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:54.303951 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:54.304976 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.305670 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.307376 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.307819 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.307920 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:54.311369 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:54.311376 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:54.311381 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:54.311428 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:54.314792 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.315435 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.315440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.315444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.315586 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.315981 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.315985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.315988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.316345 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:54.316510 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:54.316875 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 04:38:54.317097 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 04:38:54.318472 23258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } has no permanent_uuid. Determining permanent_uuid...
I20260619 04:38:54.319427 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.319060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39523 (local address 127.0.0.1:55750)
0619 04:38:54.319209 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.319282 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 04:38:54.320253 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.319954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39523 (local address 127.0.0.1:55756)
0619 04:38:54.320085 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.320147 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 04:38:54.320431 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39523: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
I20260619 04:38:54.349946 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } attempt: 3
I20260619 04:38:54.351025 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.350558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39523 (local address 127.0.0.1:55768)
0619 04:38:54.350779 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.350864 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 04:38:54.352020 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.351576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39523 (local address 127.0.0.1:55770)
0619 04:38:54.351801 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.351888 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260619 04:38:54.352210 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39523: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
I20260619 04:38:54.371660 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } attempt: 1
I20260619 04:38:54.372962 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.372330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39523 (local address 127.0.0.1:55776)
0619 04:38:54.372610 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.372758 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 04:38:54.373935 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.373629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39523 (local address 127.0.0.1:55792)
0619 04:38:54.373758 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.373826 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 04:38:54.374127 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39523: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
I20260619 04:38:54.442324 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } attempt: 2
I20260619 04:38:54.443473 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.443010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39523 (local address 127.0.0.1:55808)
0619 04:38:54.443191 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.443293 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 04:38:54.444442 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.444138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39523 (local address 127.0.0.1:55824)
0619 04:38:54.444267 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.444331 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 04:38:54.444689 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39523: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
W20260619 04:38:54.463479 23339 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 04:38:54.463792 23339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:38:54.463887 23339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:38:54.463995 23339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:38:54.471304 23339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:38:54.471401 23339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:38:54.471472 23339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:38:54.471532 23339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:38:54.471614 23339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:38:54.481696 23339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41635,127.0.97.61:42857,127.0.97.60:39523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:38:54.483001 23339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:38:54.484293 23339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:38:54.495383 23344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:38:54.495316 23345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:54.495389 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } attempt: 4
W20260619 04:38:54.495332 23347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:38:54.496423 23339 server_base.cc:1034] running on GCE node
I20260619 04:38:54.496577 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.496044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39523 (local address 127.0.0.1:55826)
0619 04:38:54.496275 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.496373 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 04:38:54.497395 23339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:38:54.497545 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.497169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39523 (local address 127.0.0.1:55828)
0619 04:38:54.497358 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.497425 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260619 04:38:54.497776 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39523: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
I20260619 04:38:54.498858 23339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:38:54.500067 23339 hybrid_clock.cc:648] HybridClock initialized: now 1781843934500008 us; error 33 us; skew 500 ppm
I20260619 04:38:54.500381 23339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:38:54 dist-test-slave-n3tq krb5kdc[23178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:38:54.503299 23339 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260619 04:38:54.504676 23339 webserver.cc:460] Webserver started at http://127.0.97.60:37571/ using document root <none> and password file <none>
I20260619 04:38:54.505352 23339 fs_manager.cc:359] Metadata directory not provided
I20260619 04:38:54.505434 23339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:38:54.505720 23339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:38:54.508603 23339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-2/data/instance:
uuid: "02722db594ce4008827a3fcf6aaeacb5"
format_stamp: "Formatted at 2026-06-19 04:38:54 on dist-test-slave-n3tq"
server_key: "9b6b73df1047cab6a908503f73381b05"
server_key_iv: "36586deed9cd148236ce8908f112d30a"
server_key_version: "encryptionkey@0"
I20260619 04:38:54.509347 23339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-2/wal/instance:
uuid: "02722db594ce4008827a3fcf6aaeacb5"
format_stamp: "Formatted at 2026-06-19 04:38:54 on dist-test-slave-n3tq"
server_key: "9b6b73df1047cab6a908503f73381b05"
server_key_iv: "36586deed9cd148236ce8908f112d30a"
server_key_version: "encryptionkey@0"
I20260619 04:38:54.514256 23339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 04:38:54.517588 23354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:38:54.518949 23339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:38:54.519106 23339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-2/data,/tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-2/wal
uuid: "02722db594ce4008827a3fcf6aaeacb5"
format_stamp: "Formatted at 2026-06-19 04:38:54 on dist-test-slave-n3tq"
server_key: "9b6b73df1047cab6a908503f73381b05"
server_key_iv: "36586deed9cd148236ce8908f112d30a"
server_key_version: "encryptionkey@0"
I20260619 04:38:54.519308 23339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:38:54.546289 23339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:38:54.550455 23339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:38:54.550700 23339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:38:54.550871 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } attempt: 3
I20260619 04:38:54.551939 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.551394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39523 (local address 127.0.0.1:55838)
0619 04:38:54.551647 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.551747 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 04:38:54.553049 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.552634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39523 (local address 127.0.0.1:55854)
0619 04:38:54.552822 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.552915 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260619 04:38:54.553292 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39523: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39523: connect: Connection refused (error 111)
I20260619 04:38:54.567312 23339 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:39523
I20260619 04:38:54.567318 23405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:39523 every 8 connection(s)
I20260619 04:38:54.568634 23339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-2/data/info.pb
I20260619 04:38:54.572261 23406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:38:54.576174 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iQRF/build/asan/bin/kudu as pid 23339
I20260619 04:38:54.576334 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iQRF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781843588881932-388-0/minicluster-data/master-2/wal/instance
I20260619 04:38:54.576764 388 external_mini_cluster.cc:1428] Setting key b14159f53a6de09c83227a155912312f
I20260619 04:38:54.579074 23406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41635 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:38:54 dist-test-slave-n3tq krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 04:38:54 dist-test-slave-n3tq krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:38:54.598776 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.583525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39990 (local address 127.0.97.62:41635)
0619 04:38:54.583762 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:54.583768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:54.584325 (+ 557us) server_negotiation.cc:407] Connection header received
0619 04:38:54.584564 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:54.584568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:54.584641 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:54.584721 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:54.586144 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.587216 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.589124 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.589699 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.589831 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:54.593447 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:54.593452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:54.593455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:54.593489 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:54.595987 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.596764 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.596768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.596771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.596899 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.597258 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.597261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.597263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.597509 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:54.597644 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:54.598187 (+ 543us) server_negotiation.cc:299] Negotiation successful
0619 04:38:54.598407 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 04:38:54.599766 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.583472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41635 (local address 127.0.0.1:39990)
0619 04:38:54.584177 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.584228 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:54.584390 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:54.584826 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:54.584834 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:54.585653 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:54.585984 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:54.585995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.587425 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.587430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:54.588977 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:54.588989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.589913 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.589919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:54.590040 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:54.590603 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:54.590628 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:54.593263 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:54.596121 (+ 2858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:54.596133 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:54.596145 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:54.596654 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:54.597025 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:54.597030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:54.597034 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:54.597156 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:54.597641 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:54.597649 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:54.597853 (+ 204us) client_negotiation.cc:769] Sending connection context
0619 04:38:54.598073 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 04:38:54.598618 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":45,"spinlock_wait_cycles":16128,"thread_start_us":294,"threads_started":1}
I20260619 04:38:54.599783 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.578565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55862 (local address 127.0.97.60:39523)
0619 04:38:54.579387 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:54.579399 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:54.579440 (+ 41us) server_negotiation.cc:407] Connection header received
0619 04:38:54.581467 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:54.581491 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:54.581932 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:54.582250 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:54.583896 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.585907 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.587660 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.588695 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.588900 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:54.593046 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:54.593054 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:54.593066 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:54.593100 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:54.595712 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.596381 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.596388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.596390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.596589 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.596947 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.596953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.596956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.597470 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:54.597761 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:54.598073 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 04:38:54.598708 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":608,"thread_start_us":405,"threads_started":1}
I20260619 04:38:54.601511 23406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:38:54 dist-test-slave-n3tq krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 19 04:38:54 dist-test-slave-n3tq krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 04:38:54.617399 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.602202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42857 (local address 127.0.0.1:43228)
0619 04:38:54.602387 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.602406 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:54.602528 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:54.602862 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:54.602868 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:54.603297 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:54.603483 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:54.603493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.604902 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.604908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:54.606318 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:54.606333 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.607323 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.607329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:54.607592 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:54.608238 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:54.608260 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:54.611414 (+ 3154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:54.614721 (+ 3307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:54.614725 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:54.614728 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:54.615219 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:54.615653 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:54.615658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:54.615661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:54.615819 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:54.616394 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:54.616400 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:54.616595 (+ 195us) client_negotiation.cc:769] Sending connection context
0619 04:38:54.616756 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 04:38:54.617044 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 04:38:54.617453 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.602312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43228 (local address 127.0.97.61:42857)
0619 04:38:54.602498 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:54.602504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:54.602523 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:38:54.602645 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:54.602650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:54.602731 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:54.602822 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:54.603644 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.604698 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.606488 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.607166 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.607284 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:54.611655 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:54.611662 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:54.611668 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:54.611711 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:54.614569 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.615356 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.615363 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.615367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.615536 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.615918 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.615924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.615927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.616256 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:54.616402 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:54.616879 (+ 477us) server_negotiation.cc:299] Negotiation successful
0619 04:38:54.617105 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 04:38:54.618337 23406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } has no permanent_uuid. Determining permanent_uuid...
I20260619 04:38:54.620620 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.603401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39994 (local address 127.0.97.62:41635)
0619 04:38:54.603549 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:54.603554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:54.603596 (+ 42us) server_negotiation.cc:407] Connection header received
0619 04:38:54.603771 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:54.603776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:54.603837 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:54.603921 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:54.605337 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.606436 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.608103 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.608729 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.608861 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:54.613982 (+ 5121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:54.613989 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:54.613994 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:54.614038 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:54.617280 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.618533 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.618538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.618541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.618696 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.619136 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.619141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.619144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.619493 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:54.619737 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:54.620058 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 04:38:54.620272 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 04:38:54 dist-test-slave-n3tq krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 04:38:54 dist-test-slave-n3tq krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 04:38:54.631115 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.619090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39523 (local address 127.0.0.1:55872)
0619 04:38:54.619258 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.619278 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:54.619370 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:54.619664 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:54.619668 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:54.620049 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:54.620224 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:54.620236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.621401 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.621406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:54.622374 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:54.622389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.623131 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.623137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:54.623351 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:54.624021 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:54.624037 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:54.626094 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:54.628907 (+ 2813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:54.628912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:54.628917 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:54.629371 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:54.629710 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:54.629714 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:54.629716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:54.629830 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:54.630400 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:54.630403 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:54.630541 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 04:38:54.630647 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 04:38:54.630850 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 04:38:54.631349 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.619289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55872 (local address 127.0.97.60:39523)
0619 04:38:54.619426 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:54.619431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:54.619445 (+ 14us) server_negotiation.cc:407] Connection header received
0619 04:38:54.619482 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:54.619485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:54.619536 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:54.619629 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:54.620363 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.621277 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.622501 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.622998 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.623090 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:54.626252 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:54.626258 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:54.626263 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:54.626297 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:54.628784 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.629491 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.629495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.629497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.629607 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.629938 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.629944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.629948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.630284 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:54.630428 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:54.630767 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 04:38:54.631014 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 04:38:54.635746 23406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02722db594ce4008827a3fcf6aaeacb5: Bootstrap starting.
I20260619 04:38:54.636564 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.622097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43234 (local address 127.0.97.61:42857)
0619 04:38:54.622243 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:54.622247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:54.622262 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:38:54.622458 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:54.622464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:54.622540 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:54.622634 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:54.623976 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.624697 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.626082 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.626727 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.626848 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:54.631344 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:54.631351 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:54.631355 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:54.631394 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:54.634045 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.634689 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.634695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.634699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.634829 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.635180 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.635186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.635189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.635474 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:54.635711 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:54.635987 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 04:38:54.636207 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 04:38:54.639246 23406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02722db594ce4008827a3fcf6aaeacb5: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:54.640385 23406 log.cc:824] T 00000000000000000000000000000000 P 02722db594ce4008827a3fcf6aaeacb5: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:54.641014 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:38:54.643116 23406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02722db594ce4008827a3fcf6aaeacb5: No bootstrap required, opened a new log
I20260619 04:38:54.643392 23233 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "46a5f9f6b6e5409989986cd1b6c56b85"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39994
I20260619 04:38:54.650180 23406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02722db594ce4008827a3fcf6aaeacb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41635 } } peers { permanent_uuid: "8d6c3bc032d2453dac5a40bad157a9d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } } peers { permanent_uuid: "02722db594ce4008827a3fcf6aaeacb5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } }
I20260619 04:38:54.650548 23406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02722db594ce4008827a3fcf6aaeacb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02722db594ce4008827a3fcf6aaeacb5, State: Initialized, Role: FOLLOWER
I20260619 04:38:54.650985 23406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02722db594ce4008827a3fcf6aaeacb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41635 } } peers { permanent_uuid: "8d6c3bc032d2453dac5a40bad157a9d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } } peers { permanent_uuid: "02722db594ce4008827a3fcf6aaeacb5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } }
I20260619 04:38:54.651890 23412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02722db594ce4008827a3fcf6aaeacb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41635 } } peers { permanent_uuid: "8d6c3bc032d2453dac5a40bad157a9d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } } peers { permanent_uuid: "02722db594ce4008827a3fcf6aaeacb5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } } }
I20260619 04:38:54.652405 23412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02722db594ce4008827a3fcf6aaeacb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:38:54.653033 23406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02722db594ce4008827a3fcf6aaeacb5 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 04:38:54.662026 23423 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02722db594ce4008827a3fcf6aaeacb5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 04:38:54.662194 23423 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 04:38:54.659583 23339 master_runner.cc:186] Error getting master registration for 127.0.97.62:41635: OK, instance_id { permanent_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" instance_seqno: 1781843933596512 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 04:38:54.670074 23339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 04:38:54.730522 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } attempt: 4
Jun 19 04:38:54 dist-test-slave-n3tq krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 04:38:54.744910 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.731274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39523 (local address 127.0.0.1:55886)
0619 04:38:54.731496 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.731519 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:54.731653 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:54.732004 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:54.732009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:54.732396 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:54.732655 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:54.732666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.733688 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.733694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:54.734629 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:54.734642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.735397 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.735403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:54.735626 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:54.736199 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:54.736217 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:54.738476 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:54.742497 (+ 4021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:54.742503 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:54.742507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:54.742866 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:54.743204 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:54.743208 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:54.743212 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:54.743351 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:54.743927 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:54.743931 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:54.744088 (+ 157us) client_negotiation.cc:769] Sending connection context
0619 04:38:54.744207 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 04:38:54.744522 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 04:38:54.744911 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.731313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55886 (local address 127.0.97.60:39523)
0619 04:38:54.731547 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:54.731553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:54.731578 (+ 25us) server_negotiation.cc:407] Connection header received
0619 04:38:54.731795 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:54.731800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:54.731865 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:54.731947 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:54.732798 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.733544 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.734774 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.735247 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.735386 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:54.738661 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:54.738669 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:54.738674 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:54.738713 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:54.742368 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.742974 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.742980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.742987 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.743103 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.743448 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.743453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.743457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.743813 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:54.743961 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:54.744319 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 04:38:54.744556 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260619 04:38:54.750988 23258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85: Bootstrap starting.
I20260619 04:38:54.755017 23258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:54.756300 23258 log.cc:824] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:54.765933 23258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85: No bootstrap required, opened a new log
I20260619 04:38:54.773694 23258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41635 } } peers { permanent_uuid: "8d6c3bc032d2453dac5a40bad157a9d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } } peers { permanent_uuid: "02722db594ce4008827a3fcf6aaeacb5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } }
I20260619 04:38:54.774113 23258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46a5f9f6b6e5409989986cd1b6c56b85, State: Initialized, Role: FOLLOWER
I20260619 04:38:54.774622 23258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41635 } } peers { permanent_uuid: "8d6c3bc032d2453dac5a40bad157a9d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } } peers { permanent_uuid: "02722db594ce4008827a3fcf6aaeacb5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } }
I20260619 04:38:54.775849 23426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41635 } } peers { permanent_uuid: "8d6c3bc032d2453dac5a40bad157a9d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } } peers { permanent_uuid: "02722db594ce4008827a3fcf6aaeacb5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } } }
I20260619 04:38:54.776503 23426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:38:54.777082 23258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:54.777176 23233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 04:38:54.777360 23233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:54.781067 23233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41635 } } peers { permanent_uuid: "8d6c3bc032d2453dac5a40bad157a9d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } } peers { permanent_uuid: "02722db594ce4008827a3fcf6aaeacb5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } }
I20260619 04:38:54.782806 23233 leader_election.cc:288] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85 [CANDIDATE]: Term 1 election: Requested vote from peers 8d6c3bc032d2453dac5a40bad157a9d4 (127.0.97.61:42857), 02722db594ce4008827a3fcf6aaeacb5 (127.0.97.60:39523)
I20260619 04:38:54.783706 23381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "02722db594ce4008827a3fcf6aaeacb5"
I20260619 04:38:54.784228 23381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02722db594ce4008827a3fcf6aaeacb5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:54.788025 23381 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 02722db594ce4008827a3fcf6aaeacb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46a5f9f6b6e5409989986cd1b6c56b85 in term 1.
I20260619 04:38:54.788743 23210 leader_election.cc:302] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02722db594ce4008827a3fcf6aaeacb5, 46a5f9f6b6e5409989986cd1b6c56b85; no voters:
I20260619 04:38:54.789652 23426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:38:54.797017 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } attempt: 5
I20260619 04:38:54.801491 23426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85 [term 1 LEADER]: Becoming Leader. State: Replica: 46a5f9f6b6e5409989986cd1b6c56b85, State: Running, Role: LEADER
I20260619 04:38:54.802173 23426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41635 } } peers { permanent_uuid: "8d6c3bc032d2453dac5a40bad157a9d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } } peers { permanent_uuid: "02722db594ce4008827a3fcf6aaeacb5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } }
W20260619 04:38:54.802978 23360 tablet.cc:2369] T 00000000000000000000000000000000 P 02722db594ce4008827a3fcf6aaeacb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 04:38:54 dist-test-slave-n3tq krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781843934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 04:38:54.818740 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.798569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55898 (local address 127.0.97.60:39523)
0619 04:38:54.798727 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:54.798732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:54.798748 (+ 16us) server_negotiation.cc:407] Connection header received
0619 04:38:54.798805 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:54.798810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:54.798887 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:54.798984 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:54.799978 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.801122 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.802877 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.803639 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.803766 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:54.809339 (+ 5573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:54.809346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:54.809350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:54.809391 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:54.813483 (+ 4092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.814969 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.815002 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.815006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.815310 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.815976 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.815983 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.815986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.816646 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:54.816885 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:54.817725 (+ 840us) server_negotiation.cc:299] Negotiation successful
0619 04:38:54.818144 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 04:38:54.820206 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.797761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39523 (local address 127.0.0.1:55898)
0619 04:38:54.798199 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 04:38:54.798219 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:38:54.798344 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:38:54.799032 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:38:54.799037 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:38:54.799592 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:38:54.799819 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:54.799831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.801260 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.801266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:54.802663 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:38:54.802682 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.804038 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.804045 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:38:54.804275 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:54.805299 (+ 1024us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:38:54.805326 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:38:54.808730 (+ 3404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:38:54.813758 (+ 5028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:54.813764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:54.813769 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:54.814718 (+ 949us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:54.815514 (+ 796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:38:54.815537 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:38:54.815540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:38:54.815764 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:38:54.817026 (+ 1262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:38:54.817031 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:38:54.817295 (+ 264us) client_negotiation.cc:769] Sending connection context
0619 04:38:54.819116 (+ 1821us) client_negotiation.cc:240] Negotiation successful
0619 04:38:54.819588 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298}
I20260619 04:38:54.826421 23332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6c3bc032d2453dac5a40bad157a9d4: Bootstrap starting.
I20260619 04:38:54.829262 23441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46a5f9f6b6e5409989986cd1b6c56b85. Latest consensus state: current_term: 1 leader_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41635 } } peers { permanent_uuid: "8d6c3bc032d2453dac5a40bad157a9d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } } peers { permanent_uuid: "02722db594ce4008827a3fcf6aaeacb5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } } }
I20260619 04:38:54.829659 23441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85 [sys.catalog]: This master's current role is: LEADER
W20260619 04:38:54.837438 23212 tablet.cc:2369] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:38:54.844143 23332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d6c3bc032d2453dac5a40bad157a9d4: Neither blocks nor log segments found. Creating new log.
I20260619 04:38:54.845534 23332 log.cc:824] T 00000000000000000000000000000000 P 8d6c3bc032d2453dac5a40bad157a9d4: Log is configured to *not* fsync() on all Append() calls
I20260619 04:38:54.832330 23191 master_runner.cc:186] Error getting master registration for 127.0.97.61:42857: OK, instance_id { permanent_uuid: "8d6c3bc032d2453dac5a40bad157a9d4" instance_seqno: 1781843934074943 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 04:38:54.848062 23191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 04:38:54.851224 23332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6c3bc032d2453dac5a40bad157a9d4: No bootstrap required, opened a new log
W20260619 04:38:54.862739 23438 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 04:38:54.862883 23438 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 04:38:54.863010 23446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:38:54.865054 23446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:38:54.898638 23332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d6c3bc032d2453dac5a40bad157a9d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41635 } } peers { permanent_uuid: "8d6c3bc032d2453dac5a40bad157a9d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } } peers { permanent_uuid: "02722db594ce4008827a3fcf6aaeacb5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } }
I20260619 04:38:54.899073 23332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d6c3bc032d2453dac5a40bad157a9d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6c3bc032d2453dac5a40bad157a9d4, State: Initialized, Role: FOLLOWER
I20260619 04:38:54.899673 23332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d6c3bc032d2453dac5a40bad157a9d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41635 } } peers { permanent_uuid: "8d6c3bc032d2453dac5a40bad157a9d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } } peers { permanent_uuid: "02722db594ce4008827a3fcf6aaeacb5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } }
I20260619 04:38:54.902659 23381 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02722db594ce4008827a3fcf6aaeacb5 [term 1 FOLLOWER]: Refusing update from remote peer 46a5f9f6b6e5409989986cd1b6c56b85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:38:54.904019 23441 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02722db594ce4008827a3fcf6aaeacb5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:38:54.914424 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.809876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40000 (local address 127.0.97.62:41635)
0619 04:38:54.810064 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:54.810070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:54.857092 (+ 47022us) server_negotiation.cc:407] Connection header received
0619 04:38:54.863095 (+ 6003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:54.863104 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:54.863184 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:54.863256 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:54.866438 (+ 3182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.867511 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.872563 (+ 5052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.873301 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.873406 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:54.885885 (+ 12479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:54.885893 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:54.885898 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:54.885945 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:54.889718 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.900025 (+ 10307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.900031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.900034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.908645 (+ 8611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.909443 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.909474 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.909478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.910200 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:54.911963 (+ 1763us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:54.912629 (+ 666us) server_negotiation.cc:299] Negotiation successful
0619 04:38:54.913581 (+ 952us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 04:38:54.924494 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.818488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43236 (local address 127.0.97.61:42857)
0619 04:38:54.818691 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:54.818698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:54.857598 (+ 38900us) server_negotiation.cc:407] Connection header received
0619 04:38:54.858151 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:54.858158 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:54.858244 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:54.858317 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:54.868483 (+ 10166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.869577 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.874730 (+ 5153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.875439 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.875541 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:54.894107 (+ 18566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:54.894113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:54.894117 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:54.894159 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:54.899825 (+ 5666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.916982 (+ 17157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.916989 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.916993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.917209 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.918673 (+ 1464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.918678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.918681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.919093 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:54.919338 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:54.923793 (+ 4455us) server_negotiation.cc:299] Negotiation successful
0619 04:38:54.924094 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 04:38:54.925556 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:54.857812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55906 (local address 127.0.97.60:39523)
0619 04:38:54.858034 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:54.858041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:54.884956 (+ 26915us) server_negotiation.cc:407] Connection header received
0619 04:38:54.885133 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:54.885140 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:54.885233 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:54.885344 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:54.887085 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.888166 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.890008 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:54.890723 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:54.890815 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:54.893752 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:38:54.893759 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:38:54.893763 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:38:54.893806 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:38:54.897965 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.917413 (+ 19448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.917419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.917422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.917595 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:38:54.919457 (+ 1862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:38:54.919462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:38:54.919465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:38:54.919822 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:38:54.920089 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:54.924905 (+ 4816us) server_negotiation.cc:299] Negotiation successful
0619 04:38:54.925186 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 04:38:54.935988 23412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02722db594ce4008827a3fcf6aaeacb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46a5f9f6b6e5409989986cd1b6c56b85. Latest consensus state: current_term: 1 leader_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41635 } } peers { permanent_uuid: "8d6c3bc032d2453dac5a40bad157a9d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } } peers { permanent_uuid: "02722db594ce4008827a3fcf6aaeacb5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } } }
I20260619 04:38:54.935667 23451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6c3bc032d2453dac5a40bad157a9d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41635 } } peers { permanent_uuid: "8d6c3bc032d2453dac5a40bad157a9d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } } peers { permanent_uuid: "02722db594ce4008827a3fcf6aaeacb5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } } }
I20260619 04:38:54.936303 23412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02722db594ce4008827a3fcf6aaeacb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:38:54.936355 23451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6c3bc032d2453dac5a40bad157a9d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:38:54.936782 23332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d6c3bc032d2453dac5a40bad157a9d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:38:54.937160 23306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d6c3bc032d2453dac5a40bad157a9d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:38:54.939992 23307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8d6c3bc032d2453dac5a40bad157a9d4"
I20260619 04:38:54.941612 23306 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8d6c3bc032d2453dac5a40bad157a9d4 [term 1 FOLLOWER]: Refusing update from remote peer 46a5f9f6b6e5409989986cd1b6c56b85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:38:54.944701 23307 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8d6c3bc032d2453dac5a40bad157a9d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46a5f9f6b6e5409989986cd1b6c56b85 in term 1.
I20260619 04:38:54.944857 23441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41635 } } peers { permanent_uuid: "8d6c3bc032d2453dac5a40bad157a9d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } } peers { permanent_uuid: "02722db594ce4008827a3fcf6aaeacb5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } } }
I20260619 04:38:54.945133 23441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:54.946663 23452 mvcc.cc:205] Tried to move back new op lower bound from 7298432757322399744 to 7298432756960231424. Current Snapshot: MvccSnapshot[applied={T|T < 7298432757322399744}]
I20260619 04:38:54.947006 23426 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6c3bc032d2453dac5a40bad157a9d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:38:54.960429 23412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02722db594ce4008827a3fcf6aaeacb5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41635 } } peers { permanent_uuid: "8d6c3bc032d2453dac5a40bad157a9d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } } peers { permanent_uuid: "02722db594ce4008827a3fcf6aaeacb5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } } }
I20260619 04:38:54.960703 23412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02722db594ce4008827a3fcf6aaeacb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:38:54.962664 23446 catalog_manager.cc:1347] Generated new cluster ID: 328fe6e46ccf453fafc9d5935fdc2d10
I20260619 04:38:54.962774 23446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:38:54.976356 23451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6c3bc032d2453dac5a40bad157a9d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46a5f9f6b6e5409989986cd1b6c56b85. Latest consensus state: current_term: 1 leader_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41635 } } peers { permanent_uuid: "8d6c3bc032d2453dac5a40bad157a9d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } } peers { permanent_uuid: "02722db594ce4008827a3fcf6aaeacb5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } } }
I20260619 04:38:54.976722 23451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6c3bc032d2453dac5a40bad157a9d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:38:54.977006 23460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6c3bc032d2453dac5a40bad157a9d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41635 } } peers { permanent_uuid: "8d6c3bc032d2453dac5a40bad157a9d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } } peers { permanent_uuid: "02722db594ce4008827a3fcf6aaeacb5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } } }
I20260619 04:38:54.977386 23460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6c3bc032d2453dac5a40bad157a9d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:38:54.983862 23426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a5f9f6b6e5409989986cd1b6c56b85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41635 } } peers { permanent_uuid: "8d6c3bc032d2453dac5a40bad157a9d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42857 } } peers { permanent_uuid: "02722db594ce4008827a3fcf6aaeacb5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39523 } } }
I20260619 04:38:54.984177 23426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85 [sys.catalog]: This master's current role is: LEADER
I20260619 04:38:54.990113 23470 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 04:38:54.990952 23446 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:38:54.992868 23446 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:38:55.024092 23446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46a5f9f6b6e5409989986cd1b6c56b85: Generated new TSK 0
I20260619 04:38:55.027804 23446 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 04:38:55.125625 23261 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:55.118258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40016 (local address 127.0.97.62:41635)
0619 04:38:55.118452 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:55.118458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:55.122345 (+ 3887us) server_negotiation.cc:407] Connection header received
0619 04:38:55.123997 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:55.124004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:55.124091 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:55.124162 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:55.125368 (+ 1206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40016: BlockingRecv error: recv got EOF from 127.0.0.1:40016 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 04:38:55.129084 23336 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:55.118184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43250 (local address 127.0.97.61:42857)
0619 04:38:55.118400 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:55.118406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:55.127039 (+ 8633us) server_negotiation.cc:407] Connection header received
0619 04:38:55.127692 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:55.127698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:55.127793 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:55.127863 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:55.128852 (+ 989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43250: BlockingRecv error: recv got EOF from 127.0.0.1:43250 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260619 04:38:55.132221 23409 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:55.123075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55920 (local address 127.0.97.60:39523)
0619 04:38:55.123277 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:55.123283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:55.129972 (+ 6689us) server_negotiation.cc:407] Connection header received
0619 04:38:55.130708 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:55.130714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:55.130825 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:55.130897 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:55.131976 (+ 1079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55920: BlockingRecv error: recv got EOF from 127.0.0.1:55920 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 04:38:55.139245 23336 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:55.136485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43264 (local address 127.0.97.61:42857)
0619 04:38:55.136650 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:55.136655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:55.137750 (+ 1095us) server_negotiation.cc:407] Connection header received
0619 04:38:55.137924 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:55.137930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:55.138098 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:55.138200 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:55.138976 (+ 776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43264: BlockingRecv error: recv got EOF from 127.0.0.1:43264 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 04:38:55.150350 23336 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:55.143164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43276 (local address 127.0.97.61:42857)
0619 04:38:55.143311 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:55.143316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:55.148968 (+ 5652us) server_negotiation.cc:407] Connection header received
0619 04:38:55.149143 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:55.149149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:55.149248 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:55.149354 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:55.150125 (+ 771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43276: BlockingRecv error: recv got EOF from 127.0.0.1:43276 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 04:38:55.157338 23336 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:55.155309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43280 (local address 127.0.97.61:42857)
0619 04:38:55.155480 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:55.155486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:55.155954 (+ 468us) server_negotiation.cc:407] Connection header received
0619 04:38:55.156150 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:55.156156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:55.156263 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:55.156358 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:55.157097 (+ 739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43280: BlockingRecv error: recv got EOF from 127.0.0.1:43280 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 04:38:55.172046 23336 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:55.169903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43288 (local address 127.0.97.61:42857)
0619 04:38:55.170114 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:55.170120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:55.170620 (+ 500us) server_negotiation.cc:407] Connection header received
0619 04:38:55.170822 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:55.170827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:55.170915 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:55.171013 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:55.171814 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43288: BlockingRecv error: recv got EOF from 127.0.0.1:43288 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260619 04:38:55.184309 23336 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:55.182710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.0.97.61:42857)
0619 04:38:55.182899 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:55.182905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:55.183137 (+ 232us) server_negotiation.cc:407] Connection header received
0619 04:38:55.183277 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:55.183283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:55.183370 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:55.183479 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:55.184088 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43304: BlockingRecv error: recv got EOF from 127.0.0.1:43304 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 04:38:55.204977 23336 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:55.203279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43310 (local address 127.0.97.61:42857)
0619 04:38:55.203486 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:55.203492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:55.203753 (+ 261us) server_negotiation.cc:407] Connection header received
0619 04:38:55.203906 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:55.203911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:55.204008 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:55.204133 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:55.204749 (+ 616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43310: BlockingRecv error: recv got EOF from 127.0.0.1:43310 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260619 04:38:55.242060 23336 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:55.240143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.0.97.61:42857)
0619 04:38:55.240366 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:55.240373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:55.240721 (+ 348us) server_negotiation.cc:407] Connection header received
0619 04:38:55.240880 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:55.240886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:55.240980 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:55.241096 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:55.241796 (+ 700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43314: BlockingRecv error: recv got EOF from 127.0.0.1:43314 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260619 04:38:55.311291 23336 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:55.309282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.0.97.61:42857)
0619 04:38:55.309532 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:55.309539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:55.309890 (+ 351us) server_negotiation.cc:407] Connection header received
0619 04:38:55.310083 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:55.310087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:55.310162 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:55.310246 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:55.311077 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43324: BlockingRecv error: recv got EOF from 127.0.0.1:43324 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260619 04:38:55.444360 23336 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:55.442499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43328 (local address 127.0.97.61:42857)
0619 04:38:55.442708 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:55.442713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:55.443160 (+ 447us) server_negotiation.cc:407] Connection header received
0619 04:38:55.443331 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:55.443335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:55.443411 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:55.443493 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:55.444102 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43328: BlockingRecv error: recv got EOF from 127.0.0.1:43328 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 04:38:55.664081 23423 catalog_manager.cc:1259] Loaded cluster ID: 328fe6e46ccf453fafc9d5935fdc2d10
I20260619 04:38:55.664179 23423 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 02722db594ce4008827a3fcf6aaeacb5: loading cluster ID for follower catalog manager: success
I20260619 04:38:55.666559 23423 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 02722db594ce4008827a3fcf6aaeacb5: acquiring CA information for follower catalog manager: success
I20260619 04:38:55.668202 23423 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 02722db594ce4008827a3fcf6aaeacb5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 04:38:55.705644 23336 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:38:55.703975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43344 (local address 127.0.97.61:42857)
0619 04:38:55.704138 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:55.704142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:55.704348 (+ 206us) server_negotiation.cc:407] Connection header received
0619 04:38:55.704558 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:55.704563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:55.704639 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:55.704719 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:38:55.705478 (+ 759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43344: BlockingRecv error: recv got EOF from 127.0.0.1:43344 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:38:55.994647 23470 catalog_manager.cc:1259] Loaded cluster ID: 328fe6e46ccf453fafc9d5935fdc2d10
I20260619 04:38:55.994782 23470 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8d6c3bc032d2453dac5a40bad157a9d4: loading cluster ID for follower catalog manager: success
I20260619 04:38:55.997416 23470 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8d6c3bc032d2453dac5a40bad157a9d4: acquiring CA information for follower catalog manager: success
I20260619 04:38:55.999297 23470 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8d6c3bc032d2453dac5a40bad157a9d4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 04:38:56.231622 23537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:56.223923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43354 (local address 127.0.97.61:42857)
0619 04:38:56.225013 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:56.225020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:56.225388 (+ 368us) server_negotiation.cc:407] Connection header received
0619 04:38:56.225594 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:56.225597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:56.225765 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:56.225870 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:56.226385 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:56.227246 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:56.229194 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:56.229682 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:56.229786 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:56.230358 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:56.230928 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:56.231031 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:56.231191 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 04:38:56.231260 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":927,"thread_start_us":761,"threads_started":1}
I20260619 04:38:56.259601 23543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:56.238302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55934 (local address 127.0.97.60:39523)
0619 04:38:56.253369 (+ 15067us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:56.253374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:56.253398 (+ 24us) server_negotiation.cc:407] Connection header received
0619 04:38:56.253500 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:56.253503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:56.253658 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:56.253842 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:56.254510 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:56.255330 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:56.257362 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:56.257763 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:56.257854 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:56.258470 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:56.258985 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:56.259098 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:56.259230 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 04:38:56.259310 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14935,"thread_start_us":14820,"threads_started":1}
I20260619 04:38:56.276947 23537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:56.266433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43370 (local address 127.0.97.61:42857)
0619 04:38:56.266597 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:56.266603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:56.268977 (+ 2374us) server_negotiation.cc:407] Connection header received
0619 04:38:56.269225 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:56.269230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:56.269340 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:56.269403 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:56.270552 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:56.271739 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:56.274054 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:56.274812 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:56.274938 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:56.275645 (+ 707us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:56.275802 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:56.275902 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:56.276388 (+ 486us) server_negotiation.cc:299] Negotiation successful
0619 04:38:56.276529 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 04:38:56.278399 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:56.266165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40024 (local address 127.0.97.62:41635)
0619 04:38:56.267156 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:56.267162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:56.267181 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:38:56.267292 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:56.267297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:56.267525 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:56.268305 (+ 780us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:56.268496 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:56.269613 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:56.271874 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:56.272505 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:56.273163 (+ 658us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:56.273376 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:56.273852 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:56.277831 (+ 3979us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:56.277889 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 04:38:56.278019 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":763,"threads_started":1}
I20260619 04:38:56.281412 23543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:38:56.269210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55938 (local address 127.0.97.60:39523)
0619 04:38:56.269664 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0619 04:38:56.269671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:38:56.274513 (+ 4842us) server_negotiation.cc:407] Connection header received
0619 04:38:56.274773 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:38:56.274779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:38:56.274888 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:38:56.274950 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:38:56.275582 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:56.276677 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:56.279464 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:38:56.280051 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:38:56.280168 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:38:56.280655 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:38:56.280792 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:38:56.280905 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 04:38:56.281055 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 04:38:56.281142 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318}
I20260619 04:38:56.282871 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 23191
I20260619 04:38:56.283844 23253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:56.423983 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 23191
I20260619 04:38:56.440790 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 23265
I20260619 04:38:56.441849 23327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:56.545519 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 23265
I20260619 04:38:56.561776 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iQRF/build/asan/bin/kudu pid 23339
I20260619 04:38:56.562865 23401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:38:56.671629 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iQRF/build/asan/bin/kudu with pid 23339
2026-06-19T04:38:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4749 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8865 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (347749 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST